[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:43:33 dist-test-slave-dzhw krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:43:33 dist-test-slave-dzhw krb5kdc[399](info): set up 2 sockets
Sep 27 10:43:33 dist-test-slave-dzhw krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 10:43:34.527027 392 mini_kdc.cc:121] Time spent starting KDC: real 2.353s user 0.007s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260927 10:43:34.717170 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.190s user 0.002s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260927 10:43:34.972355 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.162s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/test-user.keytab.
W20260927 10:43:35.588030 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.616s user 0.001s sys 0.000s
Sep 27 10:43:35 dist-test-slave-dzhw krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:43:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:43:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260927 10:43:36.105659 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.493s user 0.003s sys 0.001s
I20260927 10:43:36.106637 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39741
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:36.221136 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:36.221467 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:36.221567 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:36.225639 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:43:36.225744 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:36.225785 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:36.225824 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:43:36.225857 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:43:36.231289 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:36.232479 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:36.233583 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:36.242709 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:36.242776 412 server_base.cc:1034] running on GCE node
W20260927 10:43:36.242704 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:36.242704 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:36.243665 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:36.244874 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:36.246094 412 hybrid_clock.cc:648] HybridClock initialized: now 1790505816246006 us; error 103 us; skew 500 ppm
Sep 27 10:43:36 dist-test-slave-dzhw krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:36.261461 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:43:36.262632 412 webserver.cc:460] Webserver started at http://127.0.98.62:39375/ using document root <none> and password file <none>
I20260927 10:43:36.263249 412 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:36.263342 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:36.263603 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:36.265748 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "1df74e075196489492ab1b2f1b448e65"
format_stamp: "Formatted at 2026-09-27 10:43:36 on dist-test-slave-dzhw"
server_key: "9408662887a4e6fc8678dec5156913e4"
server_key_iv: "7b41ee4444ef63e614177e0f07af9815"
server_key_version: "encryptionkey@0"
I20260927 10:43:36.266366 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "1df74e075196489492ab1b2f1b448e65"
format_stamp: "Formatted at 2026-09-27 10:43:36 on dist-test-slave-dzhw"
server_key: "9408662887a4e6fc8678dec5156913e4"
server_key_iv: "7b41ee4444ef63e614177e0f07af9815"
server_key_version: "encryptionkey@0"
I20260927 10:43:36.270264 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 10:43:36.272816 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:36.274173 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:43:36.274333 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "1df74e075196489492ab1b2f1b448e65"
format_stamp: "Formatted at 2026-09-27 10:43:36 on dist-test-slave-dzhw"
server_key: "9408662887a4e6fc8678dec5156913e4"
server_key_iv: "7b41ee4444ef63e614177e0f07af9815"
server_key_version: "encryptionkey@0"
I20260927 10:43:36.274482 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:36.293817 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:36.297279 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:36.297508 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:36.305039 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39741
I20260927 10:43:36.305048 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39741 every 8 connection(s)
I20260927 10:43:36.306030 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:43:36.308950 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:36.313735 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 412
I20260927 10:43:36.313858 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:43:36.314620 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1df74e075196489492ab1b2f1b448e65: Bootstrap starting.
I20260927 10:43:36.315322 392 external_mini_cluster.cc:1428] Setting key be224c02ad8eccd6ac52f4ef3f4339ce
I20260927 10:43:36.317363 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1df74e075196489492ab1b2f1b448e65: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:36.318300 480 log.cc:824] T 00000000000000000000000000000000 P 1df74e075196489492ab1b2f1b448e65: Log is configured to *not* fsync() on all Append() calls
I20260927 10:43:36.320281 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1df74e075196489492ab1b2f1b448e65: No bootstrap required, opened a new log
I20260927 10:43:36.324434 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1df74e075196489492ab1b2f1b448e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df74e075196489492ab1b2f1b448e65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39741 } }
I20260927 10:43:36.324728 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1df74e075196489492ab1b2f1b448e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:36.324826 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1df74e075196489492ab1b2f1b448e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df74e075196489492ab1b2f1b448e65, State: Initialized, Role: FOLLOWER
I20260927 10:43:36.325296 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1df74e075196489492ab1b2f1b448e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df74e075196489492ab1b2f1b448e65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39741 } }
I20260927 10:43:36.325419 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1df74e075196489492ab1b2f1b448e65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:43:36.325505 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1df74e075196489492ab1b2f1b448e65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:43:36.325623 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1df74e075196489492ab1b2f1b448e65 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:36.326593 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1df74e075196489492ab1b2f1b448e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df74e075196489492ab1b2f1b448e65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39741 } }
I20260927 10:43:36.326942 480 leader_election.cc:302] T 00000000000000000000000000000000 P 1df74e075196489492ab1b2f1b448e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1df74e075196489492ab1b2f1b448e65; no voters:
I20260927 10:43:36.327246 480 leader_election.cc:288] T 00000000000000000000000000000000 P 1df74e075196489492ab1b2f1b448e65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:43:36.327386 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1df74e075196489492ab1b2f1b448e65 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:43:36.327622 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1df74e075196489492ab1b2f1b448e65 [term 1 LEADER]: Becoming Leader. State: Replica: 1df74e075196489492ab1b2f1b448e65, State: Running, Role: LEADER
I20260927 10:43:36.327973 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1df74e075196489492ab1b2f1b448e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df74e075196489492ab1b2f1b448e65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39741 } }
I20260927 10:43:36.328572 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1df74e075196489492ab1b2f1b448e65 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:43:36.329542 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1df74e075196489492ab1b2f1b448e65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1df74e075196489492ab1b2f1b448e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df74e075196489492ab1b2f1b448e65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39741 } } }
I20260927 10:43:36.329680 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1df74e075196489492ab1b2f1b448e65 [sys.catalog]: This master's current role is: LEADER
I20260927 10:43:36.329991 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1df74e075196489492ab1b2f1b448e65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1df74e075196489492ab1b2f1b448e65. Latest consensus state: current_term: 1 leader_uuid: "1df74e075196489492ab1b2f1b448e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df74e075196489492ab1b2f1b448e65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39741 } } }
I20260927 10:43:36.330081 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1df74e075196489492ab1b2f1b448e65 [sys.catalog]: This master's current role is: LEADER
I20260927 10:43:36.330511 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:43:36.334076 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 10:43:36 dist-test-slave-dzhw krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:36.340613 494 catalog_manager.cc:1347] Generated new cluster ID: 4d55efee06454c408d5fc9fab9501c60
I20260927 10:43:36.340747 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:43:36.343298 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:36.318274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40724 (local address 127.0.98.62:39741)
0927 10:43:36.318756 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:36.318776 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:36.320072 (+ 1296us) server_negotiation.cc:407] Connection header received
0927 10:43:36.323919 (+ 3847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:36.323940 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:36.324320 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:36.324683 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:36.327443 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:36.328974 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:36.330892 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:36.331511 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:36.331637 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:36.336920 (+ 5283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:36.336927 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:36.336942 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:36.336977 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:36.339540 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:36.340116 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:36.340121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:36.340125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:36.340264 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:36.340704 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:36.340707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:36.340709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:36.341226 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:36.341453 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:36.342108 (+ 655us) server_negotiation.cc:299] Negotiation successful
0927 10:43:36.342387 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":172,"threads_started":1}
I20260927 10:43:36.354810 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:43:36.355947 494 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:43:36.359925 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1df74e075196489492ab1b2f1b448e65: Generated new TSK 0
I20260927 10:43:36.360630 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 10:43:36.389257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39741
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:36.500339 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:36.500608 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:36.500651 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:36.504652 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:36.504709 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:36.504819 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:43:36.510015 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39741
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:36.511089 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:36.512109 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:36.521790 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:36.521761 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:36.521747 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:36.522080 506 server_base.cc:1034] running on GCE node
I20260927 10:43:36.522522 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:36.523175 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:36.524340 506 hybrid_clock.cc:648] HybridClock initialized: now 1790505816524307 us; error 46 us; skew 500 ppm
Sep 27 10:43:36 dist-test-slave-dzhw krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:36.527105 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 10:43:36.527999 506 webserver.cc:460] Webserver started at http://127.0.98.1:33317/ using document root <none> and password file <none>
I20260927 10:43:36.528566 506 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:36.528628 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:36.528863 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:36.531090 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "e5037e3f32b14820865fb5a5800d986b"
format_stamp: "Formatted at 2026-09-27 10:43:36 on dist-test-slave-dzhw"
server_key: "4bb05bb155bd4e1c7b2ad501a77fed77"
server_key_iv: "0c17b1aaf8db28af4ab13bd76d0308a3"
server_key_version: "encryptionkey@0"
I20260927 10:43:36.531670 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e5037e3f32b14820865fb5a5800d986b"
format_stamp: "Formatted at 2026-09-27 10:43:36 on dist-test-slave-dzhw"
server_key: "4bb05bb155bd4e1c7b2ad501a77fed77"
server_key_iv: "0c17b1aaf8db28af4ab13bd76d0308a3"
server_key_version: "encryptionkey@0"
I20260927 10:43:36.535449 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:43:36.537875 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:36.539139 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:43:36.539278 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "e5037e3f32b14820865fb5a5800d986b"
format_stamp: "Formatted at 2026-09-27 10:43:36 on dist-test-slave-dzhw"
server_key: "4bb05bb155bd4e1c7b2ad501a77fed77"
server_key_iv: "0c17b1aaf8db28af4ab13bd76d0308a3"
server_key_version: "encryptionkey@0"
I20260927 10:43:36.539387 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:36.552838 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:36.556216 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:36.556416 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:36.556977 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:43:36.557797 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:36.557864 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:36.557931 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:36.557978 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:36.566972 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39357
I20260927 10:43:36.567035 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39357 every 8 connection(s)
I20260927 10:43:36.567972 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:43:36.575603 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 506
I20260927 10:43:36.575726 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:43:36.575963 392 external_mini_cluster.cc:1428] Setting key 619a719b7f9764365100ff2b8d55c75d
Sep 27 10:43:36 dist-test-slave-dzhw krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:36.582785 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:36.569777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58201 (local address 127.0.98.62:39741)
0927 10:43:36.569943 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:36.569948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:36.570817 (+ 869us) server_negotiation.cc:407] Connection header received
0927 10:43:36.572114 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:36.572117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:36.572165 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:36.572244 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:36.573374 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:36.573949 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:36.575142 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:36.575546 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:36.575666 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:36.578508 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:36.578515 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:36.578521 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:36.578553 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:36.580430 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:36.581095 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:36.581098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:36.581099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:36.581172 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:36.581568 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:36.581570 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:36.581571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:36.581744 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:36.581859 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:36.582554 (+ 695us) server_negotiation.cc:299] Negotiation successful
0927 10:43:36.582665 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 10:43:36.584122 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:36.570174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39741 (local address 127.0.98.1:58201)
0927 10:43:36.570677 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:36.570718 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:36.571888 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:36.572399 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:36.572405 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:36.572778 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:36.573234 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:36.573247 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:36.574054 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:36.574056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:36.575020 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:36.575029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:36.575759 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:36.575766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:36.575952 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:36.576647 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:36.576664 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:36.578276 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:36.580557 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:36.580565 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:36.580586 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:36.580987 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:36.581261 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:36.581267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:36.581272 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:36.581472 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:36.581861 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:36.581871 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:36.582249 (+ 378us) client_negotiation.cc:769] Sending connection context
0927 10:43:36.582510 (+ 261us) client_negotiation.cc:240] Negotiation successful
0927 10:43:36.582915 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":111,"threads_started":1}
I20260927 10:43:36.585471 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39741
I20260927 10:43:36.585726 635 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 10:43:36.586261 635 heartbeater.cc:511] Master 127.0.98.62:39741 requested a full tablet report, sending...
I20260927 10:43:36.587870 444 ts_manager.cc:194] Registered new tserver with Master: e5037e3f32b14820865fb5a5800d986b (127.0.98.1:39357)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 10:43:36.589239 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 10:43:36.606151 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39741
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:36.715710 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:36.716101 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:36.716180 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:36.720090 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:36.720142 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:36.720201 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:43:36.725453 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39741
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:36.726490 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:36.727468 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:36.737236 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:36.737408 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:36.737236 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:36.737646 640 server_base.cc:1034] running on GCE node
I20260927 10:43:36.738090 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:36.738785 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:36.739964 640 hybrid_clock.cc:648] HybridClock initialized: now 1790505816739924 us; error 52 us; skew 500 ppm
Sep 27 10:43:36 dist-test-slave-dzhw krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:36.743044 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 10:43:36.743968 640 webserver.cc:460] Webserver started at http://127.0.98.2:35225/ using document root <none> and password file <none>
I20260927 10:43:36.744530 640 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:36.744619 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:36.744835 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:36.746939 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "7844d08bf85e42589752d093a8026d06"
format_stamp: "Formatted at 2026-09-27 10:43:36 on dist-test-slave-dzhw"
server_key: "ebc5f138188493efe4a9ee038f34e02e"
server_key_iv: "b9bef17c0778bd1ba45f115e9e84aaa7"
server_key_version: "encryptionkey@0"
I20260927 10:43:36.747519 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7844d08bf85e42589752d093a8026d06"
format_stamp: "Formatted at 2026-09-27 10:43:36 on dist-test-slave-dzhw"
server_key: "ebc5f138188493efe4a9ee038f34e02e"
server_key_iv: "b9bef17c0778bd1ba45f115e9e84aaa7"
server_key_version: "encryptionkey@0"
I20260927 10:43:36.751847 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:43:36.754251 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:36.755548 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:43:36.755784 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "7844d08bf85e42589752d093a8026d06"
format_stamp: "Formatted at 2026-09-27 10:43:36 on dist-test-slave-dzhw"
server_key: "ebc5f138188493efe4a9ee038f34e02e"
server_key_iv: "b9bef17c0778bd1ba45f115e9e84aaa7"
server_key_version: "encryptionkey@0"
I20260927 10:43:36.755896 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:36.775906 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:36.779220 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:36.779407 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:36.779958 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:43:36.780727 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:36.780789 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:36.780848 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:36.780889 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:36.789853 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43227
I20260927 10:43:36.789887 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43227 every 8 connection(s)
I20260927 10:43:36.790747 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:43:36.792595 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 640
I20260927 10:43:36.792707 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:43:36.792977 392 external_mini_cluster.cc:1428] Setting key c1efdb1232aeb9c5ce83c429a51eca04
Sep 27 10:43:36 dist-test-slave-dzhw krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 10:43:36.804275 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:36.792438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52099 (local address 127.0.98.62:39741)
0927 10:43:36.792568 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:36.792572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:36.793590 (+ 1018us) server_negotiation.cc:407] Connection header received
0927 10:43:36.795127 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:36.795130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:36.795181 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:36.795260 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:36.796393 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:36.796899 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:36.797985 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:36.798203 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:36.798310 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:36.800874 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:36.800877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:36.800880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:36.800901 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:36.802326 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:36.802794 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:36.802798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:36.802800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:36.802873 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:36.803197 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:36.803200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:36.803201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:36.803401 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:36.803501 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:36.804010 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 10:43:36.804126 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 10:43:36.805158 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:36.792901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39741 (local address 127.0.98.2:52099)
0927 10:43:36.793410 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:36.793465 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:36.794904 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:36.795435 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:36.795442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:36.795782 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:36.796254 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:36.796266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:36.797014 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:36.797018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:36.797860 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:36.797868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:36.798343 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:36.798346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:36.798546 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:36.799133 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:36.799148 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:36.800718 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:36.802458 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:36.802463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:36.802476 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:36.802700 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:36.802969 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:36.802972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:36.802974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:36.803117 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:36.803498 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:36.803503 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:36.803723 (+ 220us) client_negotiation.cc:769] Sending connection context
0927 10:43:36.803894 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 10:43:36.804179 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":138,"threads_started":1}
I20260927 10:43:36.806370 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39741
I20260927 10:43:36.806672 769 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:36.807258 769 heartbeater.cc:511] Master 127.0.98.62:39741 requested a full tablet report, sending...
I20260927 10:43:36.808415 444 ts_manager.cc:194] Registered new tserver with Master: 7844d08bf85e42589752d093a8026d06 (127.0.98.2:43227)
I20260927 10:43:36.808995 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52099
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 10:43:36.820988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39741
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:36.933406 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:36.933705 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:36.933789 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:36.937850 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:36.937948 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:36.938040 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:43:36.943434 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39741
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:36.944588 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:36.945634 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:36.955268 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:36.955348 774 server_base.cc:1034] running on GCE node
W20260927 10:43:36.955271 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:36.955272 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:36.956090 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:36.956779 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:36.957957 774 hybrid_clock.cc:648] HybridClock initialized: now 1790505816957935 us; error 33 us; skew 500 ppm
Sep 27 10:43:36 dist-test-slave-dzhw krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:36.960858 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 10:43:36.961750 774 webserver.cc:460] Webserver started at http://127.0.98.3:41939/ using document root <none> and password file <none>
I20260927 10:43:36.962322 774 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:36.962443 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:36.962690 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:36.964733 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "7ec0c0bc93794848acd726b17f27fa11"
format_stamp: "Formatted at 2026-09-27 10:43:36 on dist-test-slave-dzhw"
server_key: "ca59e27df36789d167eddcc83fea3155"
server_key_iv: "d8462ca1f6e140ef8636d9983266a90a"
server_key_version: "encryptionkey@0"
I20260927 10:43:36.965282 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7ec0c0bc93794848acd726b17f27fa11"
format_stamp: "Formatted at 2026-09-27 10:43:36 on dist-test-slave-dzhw"
server_key: "ca59e27df36789d167eddcc83fea3155"
server_key_iv: "d8462ca1f6e140ef8636d9983266a90a"
server_key_version: "encryptionkey@0"
I20260927 10:43:36.969122 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260927 10:43:36.971662 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:36.972946 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:43:36.973083 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "7ec0c0bc93794848acd726b17f27fa11"
format_stamp: "Formatted at 2026-09-27 10:43:36 on dist-test-slave-dzhw"
server_key: "ca59e27df36789d167eddcc83fea3155"
server_key_iv: "d8462ca1f6e140ef8636d9983266a90a"
server_key_version: "encryptionkey@0"
I20260927 10:43:36.973202 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:37.006654 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:37.009652 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:37.009861 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:37.010512 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:43:37.011341 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:37.011420 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:37.011492 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:37.011528 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:37.021015 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38519
I20260927 10:43:37.021039 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38519 every 8 connection(s)
I20260927 10:43:37.021898 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:43:37.027669 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 774
I20260927 10:43:37.027755 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:43:37.027994 392 external_mini_cluster.cc:1428] Setting key e073c857d94da3fb4dc7f6e215c01b7f
Sep 27 10:43:37 dist-test-slave-dzhw krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:37.035456 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:37.023833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45951 (local address 127.0.98.62:39741)
0927 10:43:37.023983 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:37.023987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:37.024798 (+ 811us) server_negotiation.cc:407] Connection header received
0927 10:43:37.026073 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:37.026076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:37.026124 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:37.026205 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:37.027536 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:37.028171 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:37.029431 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:37.029654 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:37.029763 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:37.032311 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:37.032314 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:37.032316 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:37.032338 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:37.033574 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:37.034055 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:37.034058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:37.034059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:37.034137 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:37.034508 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:37.034512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:37.034513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:37.034698 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:37.034780 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:37.035212 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 10:43:37.035309 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:43:37.036193 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:37.024213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39741 (local address 127.0.98.3:45951)
0927 10:43:37.024651 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:37.024694 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:37.025861 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:37.026396 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:37.026403 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:37.026854 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:37.027352 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:37.027366 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:37.028299 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:37.028302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:37.029308 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:37.029317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:37.029852 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:37.029859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:37.030042 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:37.030749 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:37.030765 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:37.032170 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:37.033672 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:37.033677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:37.033688 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:37.033948 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:37.034223 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:37.034225 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:37.034226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:37.034343 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:37.034793 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:37.034798 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:37.035016 (+ 218us) client_negotiation.cc:769] Sending connection context
0927 10:43:37.035166 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 10:43:37.035395 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":117,"threads_started":1}
I20260927 10:43:37.037262 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39741
I20260927 10:43:37.037520 903 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:37.037981 903 heartbeater.cc:511] Master 127.0.98.62:39741 requested a full tablet report, sending...
I20260927 10:43:37.039009 444 ts_manager.cc:194] Registered new tserver with Master: 7ec0c0bc93794848acd726b17f27fa11 (127.0.98.3:38519)
I20260927 10:43:37.039559 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45951
I20260927 10:43:37.042214 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 10:43:37 dist-test-slave-dzhw krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505817, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 10:43:37 dist-test-slave-dzhw krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505817, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 10:43:37.068135 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:37.058825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44366 (local address 127.0.98.1:39357)
0927 10:43:37.059139 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:37.059152 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:37.059176 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:43:37.059225 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:37.059229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:37.059424 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:37.059543 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:37.060247 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:37.061030 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:37.061860 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:37.062222 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:37.062315 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:37.064728 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:37.064736 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:37.064749 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:37.064771 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:37.066530 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:37.066928 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:37.066933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:37.066934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:37.067004 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:37.067221 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:37.067223 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:37.067224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:37.067485 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:37.067643 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:37.067786 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 10:43:37.067938 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":104,"threads_started":1}
W20260927 10:43:37.069142 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44366
Sep 27 10:43:37 dist-test-slave-dzhw krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 10:43:37 dist-test-slave-dzhw krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 10:43:37.094576 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:37.086213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44368 (local address 127.0.98.1:39357)
0927 10:43:37.086380 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:37.086383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:37.086404 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:43:37.086611 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:37.086614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:37.086655 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:37.086785 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:37.087474 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:37.087975 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:37.088847 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:37.089083 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:37.089163 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:37.091533 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:37.091537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:37.091539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:37.091561 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:37.093106 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:37.093496 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:37.093499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:37.093500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:37.093568 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:37.093782 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:37.093784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:37.093785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:37.093938 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:37.094033 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:37.094229 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 10:43:37.094320 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 10:43:37.095662 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 506
I20260927 10:43:37.101778 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 640
I20260927 10:43:37.109568 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 774
I20260927 10:43:37.114737 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 412
2026-09-27T10:43:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4972 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:43:37 dist-test-slave-dzhw krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:43:37 dist-test-slave-dzhw krb5kdc[917](info): set up 2 sockets
Sep 27 10:43:37 dist-test-slave-dzhw krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 10:43:38.150493 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:43:38 dist-test-slave-dzhw krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:43:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:43:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:43:38.279608 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37993
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:38.389786 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:38.390039 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:38.390080 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:38.394148 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:43:38.394207 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:38.394227 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:38.394243 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:43:38.394258 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:43:38.399555 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:38.400588 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:38.401592 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:38.409925 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:38.409884 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:38.409884 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:38.409992 930 server_base.cc:1034] running on GCE node
I20260927 10:43:38.410732 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:38.411784 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:38.412940 930 hybrid_clock.cc:648] HybridClock initialized: now 1790505818412923 us; error 30 us; skew 500 ppm
Sep 27 10:43:38 dist-test-slave-dzhw krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:38.415575 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:43:38.416445 930 webserver.cc:460] Webserver started at http://127.0.98.62:42949/ using document root <none> and password file <none>
I20260927 10:43:38.416998 930 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:38.417086 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:38.417320 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:38.419337 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "f93f8173a01a4e8ebbbc9f2cf80235f7"
format_stamp: "Formatted at 2026-09-27 10:43:38 on dist-test-slave-dzhw"
server_key: "c6ef3edc4c042031a3158b668bf9893e"
server_key_iv: "1d415d5aaaaf8439932ccb5d02c919b4"
server_key_version: "encryptionkey@0"
I20260927 10:43:38.419881 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "f93f8173a01a4e8ebbbc9f2cf80235f7"
format_stamp: "Formatted at 2026-09-27 10:43:38 on dist-test-slave-dzhw"
server_key: "c6ef3edc4c042031a3158b668bf9893e"
server_key_iv: "1d415d5aaaaf8439932ccb5d02c919b4"
server_key_version: "encryptionkey@0"
I20260927 10:43:38.423492 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:43:38.425836 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:38.427031 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:43:38.427167 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "f93f8173a01a4e8ebbbc9f2cf80235f7"
format_stamp: "Formatted at 2026-09-27 10:43:38 on dist-test-slave-dzhw"
server_key: "c6ef3edc4c042031a3158b668bf9893e"
server_key_iv: "1d415d5aaaaf8439932ccb5d02c919b4"
server_key_version: "encryptionkey@0"
I20260927 10:43:38.427277 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:38.463853 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:38.467168 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:38.467392 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:38.474323 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37993
I20260927 10:43:38.474347 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37993 every 8 connection(s)
I20260927 10:43:38.475318 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:43:38.476186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 930
I20260927 10:43:38.476305 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:43:38.476563 392 external_mini_cluster.cc:1428] Setting key ecc514f6662e0a1b893fa14ca1d3a314
I20260927 10:43:38.478888 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:38.484758 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f93f8173a01a4e8ebbbc9f2cf80235f7: Bootstrap starting.
Sep 27 10:43:38 dist-test-slave-dzhw krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:38.487490 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f93f8173a01a4e8ebbbc9f2cf80235f7: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:38.488229 998 log.cc:824] T 00000000000000000000000000000000 P f93f8173a01a4e8ebbbc9f2cf80235f7: Log is configured to *not* fsync() on all Append() calls
I20260927 10:43:38.490228 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f93f8173a01a4e8ebbbc9f2cf80235f7: No bootstrap required, opened a new log
I20260927 10:43:38.491040 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:38.478111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36912 (local address 127.0.98.62:37993)
0927 10:43:38.478660 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:38.478675 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:38.478711 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:43:38.479446 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:38.479465 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:38.479832 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:38.480197 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:38.481005 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:38.482009 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:38.482917 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:38.483390 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:38.483563 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:38.486397 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:38.486403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:38.486478 (+ 75us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:38.486508 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:38.488238 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:38.488681 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:38.488685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:38.488689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:38.488793 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:38.489096 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:38.489100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:38.489101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:38.489480 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:38.489678 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:38.489955 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 10:43:38.490235 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":182,"threads_started":1}
I20260927 10:43:38.494131 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P f93f8173a01a4e8ebbbc9f2cf80235f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93f8173a01a4e8ebbbc9f2cf80235f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } }
I20260927 10:43:38.494370 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P f93f8173a01a4e8ebbbc9f2cf80235f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:38.494529 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P f93f8173a01a4e8ebbbc9f2cf80235f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93f8173a01a4e8ebbbc9f2cf80235f7, State: Initialized, Role: FOLLOWER
I20260927 10:43:38.495035 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P f93f8173a01a4e8ebbbc9f2cf80235f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93f8173a01a4e8ebbbc9f2cf80235f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } }
I20260927 10:43:38.495195 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P f93f8173a01a4e8ebbbc9f2cf80235f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:43:38.495260 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P f93f8173a01a4e8ebbbc9f2cf80235f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:43:38.495352 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f93f8173a01a4e8ebbbc9f2cf80235f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:38.496258 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P f93f8173a01a4e8ebbbc9f2cf80235f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93f8173a01a4e8ebbbc9f2cf80235f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } }
I20260927 10:43:38.496599 998 leader_election.cc:302] T 00000000000000000000000000000000 P f93f8173a01a4e8ebbbc9f2cf80235f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f93f8173a01a4e8ebbbc9f2cf80235f7; no voters:
I20260927 10:43:38.496906 998 leader_election.cc:288] T 00000000000000000000000000000000 P f93f8173a01a4e8ebbbc9f2cf80235f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:43:38.497030 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f93f8173a01a4e8ebbbc9f2cf80235f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:43:38.497282 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P f93f8173a01a4e8ebbbc9f2cf80235f7 [term 1 LEADER]: Becoming Leader. State: Replica: f93f8173a01a4e8ebbbc9f2cf80235f7, State: Running, Role: LEADER
I20260927 10:43:38.497634 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P f93f8173a01a4e8ebbbc9f2cf80235f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93f8173a01a4e8ebbbc9f2cf80235f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } }
I20260927 10:43:38.498111 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P f93f8173a01a4e8ebbbc9f2cf80235f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:43:38.499249 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P f93f8173a01a4e8ebbbc9f2cf80235f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f93f8173a01a4e8ebbbc9f2cf80235f7. Latest consensus state: current_term: 1 leader_uuid: "f93f8173a01a4e8ebbbc9f2cf80235f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93f8173a01a4e8ebbbc9f2cf80235f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } }
I20260927 10:43:38.499377 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P f93f8173a01a4e8ebbbc9f2cf80235f7 [sys.catalog]: This master's current role is: LEADER
I20260927 10:43:38.499764 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:43:38.499732 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P f93f8173a01a4e8ebbbc9f2cf80235f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f93f8173a01a4e8ebbbc9f2cf80235f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93f8173a01a4e8ebbbc9f2cf80235f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } }
I20260927 10:43:38.499970 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P f93f8173a01a4e8ebbbc9f2cf80235f7 [sys.catalog]: This master's current role is: LEADER
I20260927 10:43:38.502796 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:43:38.508198 1012 catalog_manager.cc:1347] Generated new cluster ID: f2cdb9eabbd24d2bb76114cbcd12c38e
I20260927 10:43:38.508271 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:43:38.552619 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:43:38.553609 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:43:38.561764 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f93f8173a01a4e8ebbbc9f2cf80235f7: Generated new TSK 0
I20260927 10:43:38.562546 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 10:43:38.605357 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37993
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:38.714737 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:38.715045 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:38.715126 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:38.719208 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:38.719288 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:38.719374 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:43:38.724768 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37993
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:38.725858 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:38.726956 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:38.736817 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:38.736961 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:38.737020 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:38.736865 1024 server_base.cc:1034] running on GCE node
I20260927 10:43:38.737543 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:38.738165 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:38.739318 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790505818739303 us; error 32 us; skew 500 ppm
Sep 27 10:43:38 dist-test-slave-dzhw krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:38.742034 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 10:43:38.743129 1024 webserver.cc:460] Webserver started at http://127.0.98.1:39175/ using document root <none> and password file <none>
I20260927 10:43:38.743814 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:38.743907 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:38.744136 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:38.746174 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "0edcfed03ce24542bb6cbf4580ad6340"
format_stamp: "Formatted at 2026-09-27 10:43:38 on dist-test-slave-dzhw"
server_key: "c1e4d460de5a7e44652e294bcdd414db"
server_key_iv: "7de8b3f328215c2bff07a63423ef5b2d"
server_key_version: "encryptionkey@0"
I20260927 10:43:38.746773 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0edcfed03ce24542bb6cbf4580ad6340"
format_stamp: "Formatted at 2026-09-27 10:43:38 on dist-test-slave-dzhw"
server_key: "c1e4d460de5a7e44652e294bcdd414db"
server_key_iv: "7de8b3f328215c2bff07a63423ef5b2d"
server_key_version: "encryptionkey@0"
I20260927 10:43:38.750494 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:43:38.752854 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:38.754078 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:43:38.754215 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "0edcfed03ce24542bb6cbf4580ad6340"
format_stamp: "Formatted at 2026-09-27 10:43:38 on dist-test-slave-dzhw"
server_key: "c1e4d460de5a7e44652e294bcdd414db"
server_key_iv: "7de8b3f328215c2bff07a63423ef5b2d"
server_key_version: "encryptionkey@0"
I20260927 10:43:38.754328 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:38.767349 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:38.770291 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:38.770531 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:38.771133 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:43:38.771977 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:38.772050 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:38.772128 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:38.772171 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:38.781167 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44755
I20260927 10:43:38.781194 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44755 every 8 connection(s)
I20260927 10:43:38.782120 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:43:38.791890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 1024
I20260927 10:43:38.792017 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:43:38.792304 392 external_mini_cluster.cc:1428] Setting key ebcefe4af470546e4f040361e7fe3ef1
Sep 27 10:43:38 dist-test-slave-dzhw krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:38.799542 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:38.784114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50577 (local address 127.0.98.62:37993)
0927 10:43:38.784279 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:38.784284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:38.785205 (+ 921us) server_negotiation.cc:407] Connection header received
0927 10:43:38.787645 (+ 2440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:38.787651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:38.787710 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:38.787780 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:38.789961 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:38.790570 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:38.791752 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:38.792112 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:38.792235 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:38.795263 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:38.795266 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:38.795268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:38.795291 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:38.796960 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:38.797641 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:38.797648 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:38.797652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:38.797765 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:38.798208 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:38.798211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:38.798212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:38.798385 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:38.798555 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:38.799294 (+ 739us) server_negotiation.cc:299] Negotiation successful
0927 10:43:38.799401 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 10:43:38.800868 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:38.784365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.1:50577)
0927 10:43:38.784968 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:38.785050 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:38.787279 (+ 2229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:38.788094 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:38.788108 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:38.788772 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:38.789750 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:38.789778 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:38.790687 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:38.790690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:38.791617 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:38.791626 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:38.792272 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:38.792275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:38.792477 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:38.793159 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:38.793180 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:38.795079 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:38.797083 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:38.797093 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:38.797114 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:38.797505 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:38.797869 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:38.797875 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:38.797880 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:38.798096 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:38.798583 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:38.798591 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:38.799015 (+ 424us) client_negotiation.cc:769] Sending connection context
0927 10:43:38.799268 (+ 253us) client_negotiation.cc:240] Negotiation successful
0927 10:43:38.799668 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":112,"threads_started":1}
I20260927 10:43:38.802111 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37993
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 10:43:38.802392 1153 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:38.802899 1153 heartbeater.cc:511] Master 127.0.98.62:37993 requested a full tablet report, sending...
I20260927 10:43:38.804462 962 ts_manager.cc:194] Registered new tserver with Master: 0edcfed03ce24542bb6cbf4580ad6340 (127.0.98.1:44755)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 10:43:38.805698 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 10:43:38.821425 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37993
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:38.930181 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:38.930531 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:38.930617 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:38.934708 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:38.934777 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:38.934867 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:43:38.940112 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37993
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:38.941231 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:38.942240 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:43:38.951721 1158 server_base.cc:1034] running on GCE node
W20260927 10:43:38.951676 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:38.951682 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:38.951676 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:38.952297 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:38.952961 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:38.954157 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790505818954146 us; error 23 us; skew 500 ppm
Sep 27 10:43:38 dist-test-slave-dzhw krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:38.957017 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 10:43:38.958000 1158 webserver.cc:460] Webserver started at http://127.0.98.2:37039/ using document root <none> and password file <none>
I20260927 10:43:38.958647 1158 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:38.958736 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:38.958961 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:38.961040 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "95062353ce684c9e8ba4414dba1917ec"
format_stamp: "Formatted at 2026-09-27 10:43:38 on dist-test-slave-dzhw"
server_key: "0e719f387c0596719b6aca0d31bdc9f5"
server_key_iv: "9fd96f67e50ca181d032da356b2783aa"
server_key_version: "encryptionkey@0"
I20260927 10:43:38.961619 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "95062353ce684c9e8ba4414dba1917ec"
format_stamp: "Formatted at 2026-09-27 10:43:38 on dist-test-slave-dzhw"
server_key: "0e719f387c0596719b6aca0d31bdc9f5"
server_key_iv: "9fd96f67e50ca181d032da356b2783aa"
server_key_version: "encryptionkey@0"
I20260927 10:43:38.965375 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 10:43:38.968026 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:38.969316 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:43:38.969468 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "95062353ce684c9e8ba4414dba1917ec"
format_stamp: "Formatted at 2026-09-27 10:43:38 on dist-test-slave-dzhw"
server_key: "0e719f387c0596719b6aca0d31bdc9f5"
server_key_iv: "9fd96f67e50ca181d032da356b2783aa"
server_key_version: "encryptionkey@0"
I20260927 10:43:38.969625 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:38.988080 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:38.991389 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:38.991575 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:38.992091 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:43:38.992842 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:38.992887 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:38.992925 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:38.992940 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:39.002074 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42445
I20260927 10:43:39.002151 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42445 every 8 connection(s)
I20260927 10:43:39.003064 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:43:39.007683 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 1158
I20260927 10:43:39.007778 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:43:39.008018 392 external_mini_cluster.cc:1428] Setting key 245bb512562fbc5bb140e0271b97e3df
Sep 27 10:43:39 dist-test-slave-dzhw krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 10:43:39.017006 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:39.004862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52697 (local address 127.0.98.62:37993)
0927 10:43:39.004996 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:39.005000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:39.005813 (+ 813us) server_negotiation.cc:407] Connection header received
0927 10:43:39.007210 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:39.007213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:39.007267 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:39.007362 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:39.008669 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.009198 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.010329 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.010632 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.010743 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:39.013424 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:39.013428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:39.013431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:39.013463 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:39.014954 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:39.015472 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:39.015476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:39.015478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:39.015555 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:39.015887 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:39.015890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:39.015891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:39.016073 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:39.016178 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:39.016735 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 10:43:39.016871 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:43:39.017872 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:39.005243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.2:52697)
0927 10:43:39.005673 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:39.005715 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:39.007003 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:39.007559 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:39.007565 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:39.007967 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:39.008525 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:39.008539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.009323 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.009326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:39.010203 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:39.010211 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.010782 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.010786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:39.010954 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:39.011582 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:39.011598 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:39.013262 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:39.015082 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:39.015088 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:39.015104 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:39.015359 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:39.015644 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:39.015648 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:39.015650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:39.015785 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:39.016175 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:39.016182 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:39.016429 (+ 247us) client_negotiation.cc:769] Sending connection context
0927 10:43:39.016652 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 10:43:39.016920 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":113,"threads_started":1}
I20260927 10:43:39.019088 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 10:43:39.019316 1287 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:39.019774 1287 heartbeater.cc:511] Master 127.0.98.62:37993 requested a full tablet report, sending...
I20260927 10:43:39.020821 962 ts_manager.cc:194] Registered new tserver with Master: 95062353ce684c9e8ba4414dba1917ec (127.0.98.2:42445)
I20260927 10:43:39.021507 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 10:43:39.035354 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37993
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:39.145387 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:39.145650 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:39.145694 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:39.150345 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:39.150409 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:39.150554 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:43:39.155768 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37993
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:39.156737 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:39.157691 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:39.167279 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:39.167304 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:39.167402 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:39.167501 1292 server_base.cc:1034] running on GCE node
I20260927 10:43:39.168231 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:39.169070 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:39.170250 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790505819170236 us; error 25 us; skew 500 ppm
Sep 27 10:43:39 dist-test-slave-dzhw krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:39.172905 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 10:43:39.173820 1292 webserver.cc:460] Webserver started at http://127.0.98.3:36661/ using document root <none> and password file <none>
I20260927 10:43:39.174369 1292 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:39.174479 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:39.174695 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:39.176705 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "8bf82542d9394d189f170add811a9731"
format_stamp: "Formatted at 2026-09-27 10:43:39 on dist-test-slave-dzhw"
server_key: "523ac07fdc0bde398e96b1bdaeea8b40"
server_key_iv: "4b88256c2ac511953b36a1e566f9ca31"
server_key_version: "encryptionkey@0"
I20260927 10:43:39.177335 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8bf82542d9394d189f170add811a9731"
format_stamp: "Formatted at 2026-09-27 10:43:39 on dist-test-slave-dzhw"
server_key: "523ac07fdc0bde398e96b1bdaeea8b40"
server_key_iv: "4b88256c2ac511953b36a1e566f9ca31"
server_key_version: "encryptionkey@0"
I20260927 10:43:39.181031 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 10:43:39.183471 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:39.184804 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:43:39.184962 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "8bf82542d9394d189f170add811a9731"
format_stamp: "Formatted at 2026-09-27 10:43:39 on dist-test-slave-dzhw"
server_key: "523ac07fdc0bde398e96b1bdaeea8b40"
server_key_iv: "4b88256c2ac511953b36a1e566f9ca31"
server_key_version: "encryptionkey@0"
I20260927 10:43:39.185082 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:39.201774 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:39.204804 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:39.205022 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:39.205621 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:43:39.206571 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:39.206640 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:39.206717 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:39.206769 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:39.216421 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36491
I20260927 10:43:39.216461 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36491 every 8 connection(s)
I20260927 10:43:39.217414 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:43:39.221777 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 1292
I20260927 10:43:39.221927 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:43:39.222288 392 external_mini_cluster.cc:1428] Setting key 7810ea55f621f413a4bc9b9784c0a16a
Sep 27 10:43:39 dist-test-slave-dzhw krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:39.232156 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:39.219403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44843 (local address 127.0.98.62:37993)
0927 10:43:39.219546 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:39.219559 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:39.220403 (+ 844us) server_negotiation.cc:407] Connection header received
0927 10:43:39.222038 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:39.222042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:39.222105 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:39.222185 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:39.223910 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.224567 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.225800 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.226044 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.226165 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:39.228750 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:39.228754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:39.228756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:39.228781 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:39.230118 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:39.230740 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:39.230742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:39.230744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:39.230817 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:39.231086 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:39.231088 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:39.231089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:39.231250 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:39.231323 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:39.231932 (+ 609us) server_negotiation.cc:299] Negotiation successful
0927 10:43:39.232032 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:43:39.232981 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:39.219711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.3:44843)
0927 10:43:39.220238 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:39.220284 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:39.221725 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:39.222450 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:39.222460 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:39.222997 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:39.223746 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:39.223767 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.224701 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.224705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:39.225667 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:39.225676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.226198 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.226202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:39.226402 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:39.227058 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:39.227075 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:39.228591 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:39.230274 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:39.230280 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:39.230292 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:39.230661 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:39.230895 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:39.230897 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:39.230898 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:39.231024 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:39.231396 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:39.231418 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:39.231705 (+ 287us) client_negotiation.cc:769] Sending connection context
0927 10:43:39.231860 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 10:43:39.232101 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":123,"threads_started":1}
I20260927 10:43:39.234157 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37993
I20260927 10:43:39.234503 1421 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:39.235146 1421 heartbeater.cc:511] Master 127.0.98.62:37993 requested a full tablet report, sending...
I20260927 10:43:39.236351 961 ts_manager.cc:194] Registered new tserver with Master: 8bf82542d9394d189f170add811a9731 (127.0.98.3:36491)
I20260927 10:43:39.236949 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44843
I20260927 10:43:39.248310 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:43:39.259318 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:39.251957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36914 (local address 127.0.98.62:37993)
0927 10:43:39.252109 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:39.252112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:39.252259 (+ 147us) server_negotiation.cc:407] Connection header received
0927 10:43:39.252408 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:39.252411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:39.252465 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:39.252555 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:39.253250 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.253778 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.254740 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.255007 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.255099 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:39.256520 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:39.256523 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:39.256525 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:39.256549 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:39.257819 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:39.258310 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:39.258313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:39.258314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:39.258393 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:39.258620 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:39.258622 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:39.258623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:39.258822 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:39.258916 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:39.259094 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 10:43:39.259190 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 10:43:39.264500 961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:43:39.267560 961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:43:39.282547 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:39.276217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36491 (local address 127.0.0.1:45868)
0927 10:43:39.276625 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:39.276678 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:39.276807 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:39.278481 (+ 1674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:39.278484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:39.278500 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:43:39.278592 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:39.278597 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.279931 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.279936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:39.281070 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:39.281079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.281964 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.281967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:39.282079 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:39.282117 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 10:43:39.282225 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 10:43:39.282360 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":1,"thread_start_us":134,"threads_started":1}
I20260927 10:43:39.282552 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:39.276439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45868 (local address 127.0.98.3:36491)
0927 10:43:39.277117 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:39.277129 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:39.277154 (+ 25us) server_negotiation.cc:407] Connection header received
0927 10:43:39.277916 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:39.277922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:39.278154 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:39.278274 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:43:39.278720 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.279796 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.281190 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.281808 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.281957 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:39.282115 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:39.282266 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 10:43:39.282360 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":153,"threads_started":1}
I20260927 10:43:39.283000 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:39.276552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42445 (local address 127.0.0.1:55328)
0927 10:43:39.277022 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:39.277036 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:39.277124 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:39.278093 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:39.278101 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:39.278128 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:43:39.278314 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:39.278322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.279787 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.279790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:39.281208 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:39.281217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.282585 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.282589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:39.282685 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:39.282714 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 10:43:39.282784 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 10:43:39.282859 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":66,"thread_start_us":99,"threads_started":1}
I20260927 10:43:39.282999 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:39.276986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44755 (local address 127.0.0.1:38744)
0927 10:43:39.277317 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:39.277331 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:39.277415 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:39.278173 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:39.278176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:39.278192 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:43:39.278343 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:39.278350 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.279422 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.279425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:39.280913 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:39.280924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.282384 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.282387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:39.282539 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:39.282569 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 10:43:39.282824 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 10:43:39.282879 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":92,"thread_start_us":78,"threads_started":1}
I20260927 10:43:39.283526 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:39.276989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55328 (local address 127.0.98.2:42445)
0927 10:43:39.277371 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:39.277384 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:39.277419 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:43:39.277499 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:39.277507 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:39.277746 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:39.277900 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:43:39.278471 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.279639 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.281322 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.282264 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.282368 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:39.282528 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:39.283260 (+ 732us) server_negotiation.cc:299] Negotiation successful
0927 10:43:39.283371 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":138,"threads_started":1}
I20260927 10:43:39.283975 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:39.277388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38744 (local address 127.0.98.1:44755)
0927 10:43:39.277727 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:39.277743 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:39.277762 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:43:39.277833 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:39.277840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:39.278029 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:39.278171 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:43:39.278499 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.279296 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.281048 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.281877 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.282158 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:39.282319 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:39.283261 (+ 942us) server_negotiation.cc:299] Negotiation successful
0927 10:43:39.283836 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":157,"threads_started":1}
I20260927 10:43:39.285828 1221 tablet_service.cc:1467] Processing CreateTablet for tablet e5ba59c6c3444e598b66b3ac9cfdca40 (DEFAULT_TABLE table=test-table [id=f37a2ec948dd4ed69b886aa22ca83ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:39.285795 1355 tablet_service.cc:1467] Processing CreateTablet for tablet e5ba59c6c3444e598b66b3ac9cfdca40 (DEFAULT_TABLE table=test-table [id=f37a2ec948dd4ed69b886aa22ca83ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:39.286388 1087 tablet_service.cc:1467] Processing CreateTablet for tablet e5ba59c6c3444e598b66b3ac9cfdca40 (DEFAULT_TABLE table=test-table [id=f37a2ec948dd4ed69b886aa22ca83ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:39.286898 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ba59c6c3444e598b66b3ac9cfdca40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:39.286921 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ba59c6c3444e598b66b3ac9cfdca40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:39.287366 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ba59c6c3444e598b66b3ac9cfdca40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:39.293797 1439 tablet_bootstrap.cc:492] T e5ba59c6c3444e598b66b3ac9cfdca40 P 8bf82542d9394d189f170add811a9731: Bootstrap starting.
I20260927 10:43:39.294597 1438 tablet_bootstrap.cc:492] T e5ba59c6c3444e598b66b3ac9cfdca40 P 95062353ce684c9e8ba4414dba1917ec: Bootstrap starting.
I20260927 10:43:39.295485 1440 tablet_bootstrap.cc:492] T e5ba59c6c3444e598b66b3ac9cfdca40 P 0edcfed03ce24542bb6cbf4580ad6340: Bootstrap starting.
I20260927 10:43:39.296062 1439 tablet_bootstrap.cc:654] T e5ba59c6c3444e598b66b3ac9cfdca40 P 8bf82542d9394d189f170add811a9731: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:39.296574 1438 tablet_bootstrap.cc:654] T e5ba59c6c3444e598b66b3ac9cfdca40 P 95062353ce684c9e8ba4414dba1917ec: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:39.297416 1439 log.cc:824] T e5ba59c6c3444e598b66b3ac9cfdca40 P 8bf82542d9394d189f170add811a9731: Log is configured to *not* fsync() on all Append() calls
I20260927 10:43:39.297978 1440 tablet_bootstrap.cc:654] T e5ba59c6c3444e598b66b3ac9cfdca40 P 0edcfed03ce24542bb6cbf4580ad6340: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:39.298187 1438 log.cc:824] T e5ba59c6c3444e598b66b3ac9cfdca40 P 95062353ce684c9e8ba4414dba1917ec: Log is configured to *not* fsync() on all Append() calls
I20260927 10:43:39.299305 1440 log.cc:824] T e5ba59c6c3444e598b66b3ac9cfdca40 P 0edcfed03ce24542bb6cbf4580ad6340: Log is configured to *not* fsync() on all Append() calls
I20260927 10:43:39.300042 1439 tablet_bootstrap.cc:492] T e5ba59c6c3444e598b66b3ac9cfdca40 P 8bf82542d9394d189f170add811a9731: No bootstrap required, opened a new log
I20260927 10:43:39.300247 1439 ts_tablet_manager.cc:1397] T e5ba59c6c3444e598b66b3ac9cfdca40 P 8bf82542d9394d189f170add811a9731: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 10:43:39.300256 1438 tablet_bootstrap.cc:492] T e5ba59c6c3444e598b66b3ac9cfdca40 P 95062353ce684c9e8ba4414dba1917ec: No bootstrap required, opened a new log
I20260927 10:43:39.300462 1438 ts_tablet_manager.cc:1397] T e5ba59c6c3444e598b66b3ac9cfdca40 P 95062353ce684c9e8ba4414dba1917ec: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:43:39.301205 1440 tablet_bootstrap.cc:492] T e5ba59c6c3444e598b66b3ac9cfdca40 P 0edcfed03ce24542bb6cbf4580ad6340: No bootstrap required, opened a new log
I20260927 10:43:39.301381 1440 ts_tablet_manager.cc:1397] T e5ba59c6c3444e598b66b3ac9cfdca40 P 0edcfed03ce24542bb6cbf4580ad6340: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 10:43:39.303508 1438 raft_consensus.cc:348] T e5ba59c6c3444e598b66b3ac9cfdca40 P 95062353ce684c9e8ba4414dba1917ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf82542d9394d189f170add811a9731" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36491 } } peers { permanent_uuid: "0edcfed03ce24542bb6cbf4580ad6340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44755 } } peers { permanent_uuid: "95062353ce684c9e8ba4414dba1917ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42445 } }
I20260927 10:43:39.303773 1438 raft_consensus.cc:374] T e5ba59c6c3444e598b66b3ac9cfdca40 P 95062353ce684c9e8ba4414dba1917ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:39.303831 1438 raft_consensus.cc:729] T e5ba59c6c3444e598b66b3ac9cfdca40 P 95062353ce684c9e8ba4414dba1917ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95062353ce684c9e8ba4414dba1917ec, State: Initialized, Role: FOLLOWER
I20260927 10:43:39.304087 1439 raft_consensus.cc:348] T e5ba59c6c3444e598b66b3ac9cfdca40 P 8bf82542d9394d189f170add811a9731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf82542d9394d189f170add811a9731" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36491 } } peers { permanent_uuid: "0edcfed03ce24542bb6cbf4580ad6340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44755 } } peers { permanent_uuid: "95062353ce684c9e8ba4414dba1917ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42445 } }
I20260927 10:43:39.304345 1439 raft_consensus.cc:374] T e5ba59c6c3444e598b66b3ac9cfdca40 P 8bf82542d9394d189f170add811a9731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:39.304276 1438 consensus_queue.cc:260] T e5ba59c6c3444e598b66b3ac9cfdca40 P 95062353ce684c9e8ba4414dba1917ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf82542d9394d189f170add811a9731" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36491 } } peers { permanent_uuid: "0edcfed03ce24542bb6cbf4580ad6340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44755 } } peers { permanent_uuid: "95062353ce684c9e8ba4414dba1917ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42445 } }
I20260927 10:43:39.304404 1439 raft_consensus.cc:729] T e5ba59c6c3444e598b66b3ac9cfdca40 P 8bf82542d9394d189f170add811a9731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf82542d9394d189f170add811a9731, State: Initialized, Role: FOLLOWER
I20260927 10:43:39.305042 1287 heartbeater.cc:503] Master 127.0.98.62:37993 was elected leader, sending a full tablet report...
I20260927 10:43:39.305044 1440 raft_consensus.cc:348] T e5ba59c6c3444e598b66b3ac9cfdca40 P 0edcfed03ce24542bb6cbf4580ad6340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf82542d9394d189f170add811a9731" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36491 } } peers { permanent_uuid: "0edcfed03ce24542bb6cbf4580ad6340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44755 } } peers { permanent_uuid: "95062353ce684c9e8ba4414dba1917ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42445 } }
I20260927 10:43:39.305279 1440 raft_consensus.cc:374] T e5ba59c6c3444e598b66b3ac9cfdca40 P 0edcfed03ce24542bb6cbf4580ad6340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:39.305337 1440 raft_consensus.cc:729] T e5ba59c6c3444e598b66b3ac9cfdca40 P 0edcfed03ce24542bb6cbf4580ad6340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0edcfed03ce24542bb6cbf4580ad6340, State: Initialized, Role: FOLLOWER
I20260927 10:43:39.305382 1438 ts_tablet_manager.cc:1428] T e5ba59c6c3444e598b66b3ac9cfdca40 P 95062353ce684c9e8ba4414dba1917ec: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:43:39.304798 1439 consensus_queue.cc:260] T e5ba59c6c3444e598b66b3ac9cfdca40 P 8bf82542d9394d189f170add811a9731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf82542d9394d189f170add811a9731" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36491 } } peers { permanent_uuid: "0edcfed03ce24542bb6cbf4580ad6340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44755 } } peers { permanent_uuid: "95062353ce684c9e8ba4414dba1917ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42445 } }
I20260927 10:43:39.306046 1421 heartbeater.cc:503] Master 127.0.98.62:37993 was elected leader, sending a full tablet report...
I20260927 10:43:39.306349 1439 ts_tablet_manager.cc:1428] T e5ba59c6c3444e598b66b3ac9cfdca40 P 8bf82542d9394d189f170add811a9731: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:43:39.305765 1440 consensus_queue.cc:260] T e5ba59c6c3444e598b66b3ac9cfdca40 P 0edcfed03ce24542bb6cbf4580ad6340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf82542d9394d189f170add811a9731" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36491 } } peers { permanent_uuid: "0edcfed03ce24542bb6cbf4580ad6340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44755 } } peers { permanent_uuid: "95062353ce684c9e8ba4414dba1917ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42445 } }
I20260927 10:43:39.307370 1153 heartbeater.cc:503] Master 127.0.98.62:37993 was elected leader, sending a full tablet report...
I20260927 10:43:39.307385 1440 ts_tablet_manager.cc:1428] T e5ba59c6c3444e598b66b3ac9cfdca40 P 0edcfed03ce24542bb6cbf4580ad6340: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260927 10:43:39.386795 1444 raft_consensus.cc:482] T e5ba59c6c3444e598b66b3ac9cfdca40 P 95062353ce684c9e8ba4414dba1917ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:43:39.387059 1444 raft_consensus.cc:504] T e5ba59c6c3444e598b66b3ac9cfdca40 P 95062353ce684c9e8ba4414dba1917ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf82542d9394d189f170add811a9731" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36491 } } peers { permanent_uuid: "0edcfed03ce24542bb6cbf4580ad6340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44755 } } peers { permanent_uuid: "95062353ce684c9e8ba4414dba1917ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42445 } }
I20260927 10:43:39.388337 1444 leader_election.cc:288] T e5ba59c6c3444e598b66b3ac9cfdca40 P 95062353ce684c9e8ba4414dba1917ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bf82542d9394d189f170add811a9731 (127.0.98.3:36491), 0edcfed03ce24542bb6cbf4580ad6340 (127.0.98.1:44755)
I20260927 10:43:39.392531 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:39.388648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36491 (local address 127.0.98.2:37823)
0927 10:43:39.388798 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:39.388814 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:39.388918 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:39.389317 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:39.389320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:39.389340 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:43:39.389495 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:39.389503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.390369 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.390373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:39.391290 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:39.391299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.392069 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.392073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:39.392185 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:39.392226 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 10:43:39.392296 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 10:43:39.392374 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 10:43:39.392530 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:39.388723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37823 (local address 127.0.98.3:36491)
0927 10:43:39.388866 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:39.388876 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:39.388892 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:43:39.389056 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:39.389062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:39.389142 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:39.389237 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:43:39.389595 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.390258 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.391421 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.391945 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.392036 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:39.392088 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:39.392321 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 10:43:39.392388 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 10:43:39.392750 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:39.389135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45643 (local address 127.0.98.1:44755)
0927 10:43:39.389278 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:39.389285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:39.389437 (+ 152us) server_negotiation.cc:407] Connection header received
0927 10:43:39.389538 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:39.389544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:39.389604 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:39.389672 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:43:39.389995 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.390593 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.391715 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.392153 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.392257 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:39.392325 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:39.392556 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 10:43:39.392632 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 10:43:39.392895 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:39.388985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44755 (local address 127.0.98.2:45643)
0927 10:43:39.389328 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:39.389354 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:39.389443 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:39.389705 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:39.389707 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:39.389725 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:43:39.389859 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:39.389865 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.390717 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.390720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:39.391588 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:39.391598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.392284 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.392290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:39.392388 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:39.392438 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 10:43:39.392514 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 10:43:39.392725 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":128,"threads_started":1}
I20260927 10:43:39.393258 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ba59c6c3444e598b66b3ac9cfdca40" candidate_uuid: "95062353ce684c9e8ba4414dba1917ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf82542d9394d189f170add811a9731" is_pre_election: true
I20260927 10:43:39.393580 1375 raft_consensus.cc:2454] T e5ba59c6c3444e598b66b3ac9cfdca40 P 8bf82542d9394d189f170add811a9731 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95062353ce684c9e8ba4414dba1917ec in term 0.
I20260927 10:43:39.394112 1174 leader_election.cc:302] T e5ba59c6c3444e598b66b3ac9cfdca40 P 95062353ce684c9e8ba4414dba1917ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bf82542d9394d189f170add811a9731, 95062353ce684c9e8ba4414dba1917ec; no voters:
I20260927 10:43:39.394352 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ba59c6c3444e598b66b3ac9cfdca40" candidate_uuid: "95062353ce684c9e8ba4414dba1917ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0edcfed03ce24542bb6cbf4580ad6340" is_pre_election: true
I20260927 10:43:39.394451 1444 raft_consensus.cc:2781] T e5ba59c6c3444e598b66b3ac9cfdca40 P 95062353ce684c9e8ba4414dba1917ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:43:39.394527 1444 raft_consensus.cc:482] T e5ba59c6c3444e598b66b3ac9cfdca40 P 95062353ce684c9e8ba4414dba1917ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:43:39.394594 1444 raft_consensus.cc:3037] T e5ba59c6c3444e598b66b3ac9cfdca40 P 95062353ce684c9e8ba4414dba1917ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:39.394717 1107 raft_consensus.cc:2454] T e5ba59c6c3444e598b66b3ac9cfdca40 P 0edcfed03ce24542bb6cbf4580ad6340 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95062353ce684c9e8ba4414dba1917ec in term 0.
I20260927 10:43:39.395815 1444 raft_consensus.cc:504] T e5ba59c6c3444e598b66b3ac9cfdca40 P 95062353ce684c9e8ba4414dba1917ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf82542d9394d189f170add811a9731" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36491 } } peers { permanent_uuid: "0edcfed03ce24542bb6cbf4580ad6340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44755 } } peers { permanent_uuid: "95062353ce684c9e8ba4414dba1917ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42445 } }
I20260927 10:43:39.396193 1444 leader_election.cc:288] T e5ba59c6c3444e598b66b3ac9cfdca40 P 95062353ce684c9e8ba4414dba1917ec [CANDIDATE]: Term 1 election: Requested vote from peers 8bf82542d9394d189f170add811a9731 (127.0.98.3:36491), 0edcfed03ce24542bb6cbf4580ad6340 (127.0.98.1:44755)
I20260927 10:43:39.396571 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ba59c6c3444e598b66b3ac9cfdca40" candidate_uuid: "95062353ce684c9e8ba4414dba1917ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf82542d9394d189f170add811a9731"
I20260927 10:43:39.396682 1375 raft_consensus.cc:3037] T e5ba59c6c3444e598b66b3ac9cfdca40 P 8bf82542d9394d189f170add811a9731 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:39.396705 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ba59c6c3444e598b66b3ac9cfdca40" candidate_uuid: "95062353ce684c9e8ba4414dba1917ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0edcfed03ce24542bb6cbf4580ad6340"
I20260927 10:43:39.396843 1107 raft_consensus.cc:3037] T e5ba59c6c3444e598b66b3ac9cfdca40 P 0edcfed03ce24542bb6cbf4580ad6340 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:39.398170 1107 raft_consensus.cc:2454] T e5ba59c6c3444e598b66b3ac9cfdca40 P 0edcfed03ce24542bb6cbf4580ad6340 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95062353ce684c9e8ba4414dba1917ec in term 1.
I20260927 10:43:39.398501 1174 leader_election.cc:302] T e5ba59c6c3444e598b66b3ac9cfdca40 P 95062353ce684c9e8ba4414dba1917ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0edcfed03ce24542bb6cbf4580ad6340, 95062353ce684c9e8ba4414dba1917ec; no voters:
I20260927 10:43:39.398556 1375 raft_consensus.cc:2454] T e5ba59c6c3444e598b66b3ac9cfdca40 P 8bf82542d9394d189f170add811a9731 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95062353ce684c9e8ba4414dba1917ec in term 1.
I20260927 10:43:39.398727 1444 raft_consensus.cc:2781] T e5ba59c6c3444e598b66b3ac9cfdca40 P 95062353ce684c9e8ba4414dba1917ec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:43:39.398988 1444 raft_consensus.cc:686] T e5ba59c6c3444e598b66b3ac9cfdca40 P 95062353ce684c9e8ba4414dba1917ec [term 1 LEADER]: Becoming Leader. State: Replica: 95062353ce684c9e8ba4414dba1917ec, State: Running, Role: LEADER
I20260927 10:43:39.399392 1444 consensus_queue.cc:237] T e5ba59c6c3444e598b66b3ac9cfdca40 P 95062353ce684c9e8ba4414dba1917ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf82542d9394d189f170add811a9731" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36491 } } peers { permanent_uuid: "0edcfed03ce24542bb6cbf4580ad6340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44755 } } peers { permanent_uuid: "95062353ce684c9e8ba4414dba1917ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42445 } }
I20260927 10:43:39.402869 962 catalog_manager.cc:5495] T e5ba59c6c3444e598b66b3ac9cfdca40 P 95062353ce684c9e8ba4414dba1917ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 95062353ce684c9e8ba4414dba1917ec (127.0.98.2). New cstate: current_term: 1 leader_uuid: "95062353ce684c9e8ba4414dba1917ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf82542d9394d189f170add811a9731" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0edcfed03ce24542bb6cbf4580ad6340" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95062353ce684c9e8ba4414dba1917ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42445 } health_report { overall_health: HEALTHY } } }
Sep 27 10:43:39 dist-test-slave-dzhw krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 10:43:39.438649 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:39.429613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38748 (local address 127.0.98.1:44755)
0927 10:43:39.429778 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:39.429782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:39.429908 (+ 126us) server_negotiation.cc:407] Connection header received
0927 10:43:39.430091 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:39.430094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:39.430161 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:39.430296 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:39.431063 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.431572 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.432402 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.432625 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.432737 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:39.435319 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:39.435331 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:39.435347 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:39.435370 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:39.437030 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:39.437436 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:39.437441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:39.437443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:39.437523 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:39.437759 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:39.437765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:39.437767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:39.438032 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:39.438190 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:39.438365 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 10:43:39.438515 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 10:43:39 dist-test-slave-dzhw krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505819, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 10:43:39 dist-test-slave-dzhw krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505819, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 10:43:39.466653 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:39.457657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38750 (local address 127.0.98.1:44755)
0927 10:43:39.457793 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:39.457798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:39.457974 (+ 176us) server_negotiation.cc:407] Connection header received
0927 10:43:39.458137 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:39.458140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:39.458181 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:39.458271 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:39.459041 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.459602 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.460440 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.460673 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.460780 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:39.463307 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:39.463310 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:39.463312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:39.463341 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:39.464832 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:39.465249 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:39.465252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:39.465253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:39.465331 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:39.465594 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:39.465597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:39.465598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:39.465777 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:39.465919 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:39.466244 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 10:43:39.466403 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 10:43:39.468468 1422 tablet.cc:2369] T e5ba59c6c3444e598b66b3ac9cfdca40 P 8bf82542d9394d189f170add811a9731: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 10:43:39 dist-test-slave-dzhw krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 10:43:39 dist-test-slave-dzhw krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 10:43:39.496738 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:39.485878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38758 (local address 127.0.98.1:44755)
0927 10:43:39.486016 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:39.486021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:39.486264 (+ 243us) server_negotiation.cc:407] Connection header received
0927 10:43:39.486549 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:39.486555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:39.486617 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:39.486706 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:39.487753 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.488627 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.490086 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:39.490524 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:39.490663 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:39.493562 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:39.493566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:39.493569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:39.493605 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:39.495072 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:39.495518 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:39.495523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:39.495525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:39.495606 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:39.495886 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:39.495890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:39.495894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:39.496082 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:39.496214 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:39.496457 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 10:43:39.496573 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 10:43:39.498337 1067 tablet_service.cc:2908] Rejecting scan request for tablet e5ba59c6c3444e598b66b3ac9cfdca40: Uninitialized: clean time has not yet been initialized
I20260927 10:43:39.500450 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 1024
W20260927 10:43:39.504312 1288 tablet.cc:2369] T e5ba59c6c3444e598b66b3ac9cfdca40 P 95062353ce684c9e8ba4414dba1917ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:43:39.506690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 1158
I20260927 10:43:39.512383 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 1292
I20260927 10:43:39.517802 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 930
2026-09-27T10:43:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2403 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:43:39 dist-test-slave-dzhw krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:43:39 dist-test-slave-dzhw krb5kdc[1462](info): set up 2 sockets
Sep 27 10:43:39 dist-test-slave-dzhw krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260927 10:43:40.554196 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:43:40 dist-test-slave-dzhw krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:43:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:43:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:43:40.681304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38963
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:40.790307 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:40.790692 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:40.790774 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:40.794816 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:43:40.794929 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 10:43:40.794960 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:40.795001 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:40.795061 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:43:40.795099 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:43:40.800490 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38963
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:40.801659 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:40.802738 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:40.811542 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:40.811553 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:40.811616 1475 server_base.cc:1034] running on GCE node
W20260927 10:43:40.811553 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:40.812633 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:40.813761 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:40.814965 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790505820814934 us; error 43 us; skew 500 ppm
Sep 27 10:43:40 dist-test-slave-dzhw krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:40.826740 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:43:40.828610 1475 webserver.cc:460] Webserver started at http://127.0.98.62:36727/ using document root <none> and password file <none>
I20260927 10:43:40.829798 1475 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:40.829916 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:40.830449 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:40.834125 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "49f0b9e4b209470098b3ce18bd7c1de2"
format_stamp: "Formatted at 2026-09-27 10:43:40 on dist-test-slave-dzhw"
server_key: "96eebc9ed5792a90e9bd0da3b840c2ac"
server_key_iv: "ae7cfef4fcdfbf1dcb49d615a72ddcd3"
server_key_version: "encryptionkey@0"
I20260927 10:43:40.835279 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "49f0b9e4b209470098b3ce18bd7c1de2"
format_stamp: "Formatted at 2026-09-27 10:43:40 on dist-test-slave-dzhw"
server_key: "96eebc9ed5792a90e9bd0da3b840c2ac"
server_key_iv: "ae7cfef4fcdfbf1dcb49d615a72ddcd3"
server_key_version: "encryptionkey@0"
I20260927 10:43:40.840620 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 10:43:40.844219 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:40.846150 1475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 10:43:40.846295 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "49f0b9e4b209470098b3ce18bd7c1de2"
format_stamp: "Formatted at 2026-09-27 10:43:40 on dist-test-slave-dzhw"
server_key: "96eebc9ed5792a90e9bd0da3b840c2ac"
server_key_iv: "ae7cfef4fcdfbf1dcb49d615a72ddcd3"
server_key_version: "encryptionkey@0"
I20260927 10:43:40.846477 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:40.865018 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:40.868566 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:40.868815 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:40.877417 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38963 every 8 connection(s)
I20260927 10:43:40.877415 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38963
I20260927 10:43:40.878557 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:43:40.881376 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:40.887153 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f0b9e4b209470098b3ce18bd7c1de2: Bootstrap starting.
I20260927 10:43:40.887934 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 1475
I20260927 10:43:40.888031 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:43:40.888316 392 external_mini_cluster.cc:1428] Setting key bcc496b4ff5300bac3972789926ae886
I20260927 10:43:40.890034 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49f0b9e4b209470098b3ce18bd7c1de2: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:40.890865 1553 log.cc:824] T 00000000000000000000000000000000 P 49f0b9e4b209470098b3ce18bd7c1de2: Log is configured to *not* fsync() on all Append() calls
I20260927 10:43:40.893090 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f0b9e4b209470098b3ce18bd7c1de2: No bootstrap required, opened a new log
I20260927 10:43:40.897078 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49f0b9e4b209470098b3ce18bd7c1de2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f0b9e4b209470098b3ce18bd7c1de2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38963 } }
I20260927 10:43:40.897360 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49f0b9e4b209470098b3ce18bd7c1de2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:40.897442 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49f0b9e4b209470098b3ce18bd7c1de2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f0b9e4b209470098b3ce18bd7c1de2, State: Initialized, Role: FOLLOWER
Sep 27 10:43:40 dist-test-slave-dzhw krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:40.897946 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49f0b9e4b209470098b3ce18bd7c1de2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f0b9e4b209470098b3ce18bd7c1de2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38963 } }
I20260927 10:43:40.898100 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49f0b9e4b209470098b3ce18bd7c1de2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:43:40.898172 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49f0b9e4b209470098b3ce18bd7c1de2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:43:40.898293 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49f0b9e4b209470098b3ce18bd7c1de2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:40.899551 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49f0b9e4b209470098b3ce18bd7c1de2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f0b9e4b209470098b3ce18bd7c1de2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38963 } }
I20260927 10:43:40.899904 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 49f0b9e4b209470098b3ce18bd7c1de2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49f0b9e4b209470098b3ce18bd7c1de2; no voters:
I20260927 10:43:40.900230 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 49f0b9e4b209470098b3ce18bd7c1de2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:43:40.900377 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49f0b9e4b209470098b3ce18bd7c1de2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:43:40.900599 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49f0b9e4b209470098b3ce18bd7c1de2 [term 1 LEADER]: Becoming Leader. State: Replica: 49f0b9e4b209470098b3ce18bd7c1de2, State: Running, Role: LEADER
I20260927 10:43:40.900939 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49f0b9e4b209470098b3ce18bd7c1de2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f0b9e4b209470098b3ce18bd7c1de2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38963 } }
I20260927 10:43:40.901597 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49f0b9e4b209470098b3ce18bd7c1de2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:43:40.902207 1558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f0b9e4b209470098b3ce18bd7c1de2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49f0b9e4b209470098b3ce18bd7c1de2. Latest consensus state: current_term: 1 leader_uuid: "49f0b9e4b209470098b3ce18bd7c1de2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f0b9e4b209470098b3ce18bd7c1de2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38963 } } }
I20260927 10:43:40.902320 1558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f0b9e4b209470098b3ce18bd7c1de2 [sys.catalog]: This master's current role is: LEADER
I20260927 10:43:40.902707 1563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:43:40.904235 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f0b9e4b209470098b3ce18bd7c1de2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49f0b9e4b209470098b3ce18bd7c1de2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f0b9e4b209470098b3ce18bd7c1de2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38963 } } }
I20260927 10:43:40.904336 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:40.889498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46842 (local address 127.0.98.62:38963)
0927 10:43:40.889940 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:40.889957 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:40.889992 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:43:40.890771 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:40.890791 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:40.891188 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:40.891478 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:40.892228 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:40.893635 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:40.894575 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:40.895370 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:40.895536 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:40.898188 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:40.898197 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:40.898215 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:40.898250 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:40.900196 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:40.900792 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:40.900798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:40.900804 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:40.900937 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:40.901244 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:40.901247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:40.901249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:40.901763 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:40.901990 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:40.902632 (+ 642us) server_negotiation.cc:299] Negotiation successful
0927 10:43:40.903033 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":133,"threads_started":1}
I20260927 10:43:40.904356 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f0b9e4b209470098b3ce18bd7c1de2 [sys.catalog]: This master's current role is: LEADER
I20260927 10:43:40.905656 1563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:43:40.911173 1563 catalog_manager.cc:1347] Generated new cluster ID: 6d5234e838b5401e9371de364a9a3054
I20260927 10:43:40.911252 1563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:43:40.929526 1563 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:43:40.930657 1563 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:43:40.936164 1563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49f0b9e4b209470098b3ce18bd7c1de2: Generated new TSK 0
I20260927 10:43:40.937052 1563 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 10:43:40.973630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38963
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:41.085312 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:41.085640 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:41.085716 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:41.089867 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:41.089959 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:41.090032 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:43:41.090113 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:43:41.095445 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:41.096508 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:41.097525 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:41.107170 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:41.107280 1579 server_base.cc:1034] running on GCE node
W20260927 10:43:41.107218 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:41.107184 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:41.107813 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:41.108423 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:41.109627 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790505821109583 us; error 59 us; skew 500 ppm
Sep 27 10:43:41 dist-test-slave-dzhw krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:41.112294 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 10:43:41.113204 1579 webserver.cc:460] Webserver started at http://127.0.98.1:46591/ using document root <none> and password file <none>
I20260927 10:43:41.113713 1579 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:41.113771 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:41.113933 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:41.115938 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "c602ab6ddaa948fba950c099a537742e"
format_stamp: "Formatted at 2026-09-27 10:43:41 on dist-test-slave-dzhw"
server_key: "b36bda880afd12a813dd075f85d96c03"
server_key_iv: "a343dfc83a3107325a0c6cefedd22a8c"
server_key_version: "encryptionkey@0"
I20260927 10:43:41.116422 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c602ab6ddaa948fba950c099a537742e"
format_stamp: "Formatted at 2026-09-27 10:43:41 on dist-test-slave-dzhw"
server_key: "b36bda880afd12a813dd075f85d96c03"
server_key_iv: "a343dfc83a3107325a0c6cefedd22a8c"
server_key_version: "encryptionkey@0"
I20260927 10:43:41.119951 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:43:41.122198 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:41.123411 1579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:43:41.123548 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "c602ab6ddaa948fba950c099a537742e"
format_stamp: "Formatted at 2026-09-27 10:43:41 on dist-test-slave-dzhw"
server_key: "b36bda880afd12a813dd075f85d96c03"
server_key_iv: "a343dfc83a3107325a0c6cefedd22a8c"
server_key_version: "encryptionkey@0"
I20260927 10:43:41.123649 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:41.163321 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:41.166514 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:41.166725 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:41.168213 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:41.168263 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:41.168303 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:41.168361 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:43:41 dist-test-slave-dzhw krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:41.179728 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40963
I20260927 10:43:41.179802 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40963 every 8 connection(s)
I20260927 10:43:41.180649 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:43:41.182343 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.169233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60269 (local address 127.0.98.62:38963)
0927 10:43:41.169431 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:41.169437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:41.170279 (+ 842us) server_negotiation.cc:407] Connection header received
0927 10:43:41.171497 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:41.171501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:41.171569 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:41.171678 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:41.172940 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.173478 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.174791 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.175125 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.175249 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.178550 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:41.178553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:41.178556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:41.178591 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:41.180270 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:41.180813 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:41.180816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:41.180818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:41.180907 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:41.181311 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:41.181314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:41.181316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:41.181549 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:41.181648 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:41.182106 (+ 458us) server_negotiation.cc:299] Negotiation successful
0927 10:43:41.182220 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 10:43:41.183146 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.169643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38963 (local address 127.0.98.1:60269)
0927 10:43:41.170097 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:41.170155 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:41.171233 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:41.171858 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:41.171886 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:41.172308 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:41.172787 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:41.172799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.173608 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.173611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:41.174667 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:41.174677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.175298 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.175302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:41.175475 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.176133 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:41.176150 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:41.178336 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:41.180386 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:41.180394 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:41.180410 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:41.180717 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:41.181002 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:41.181005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:41.181007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:41.181200 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:41.181656 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:41.181662 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:41.181883 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 10:43:41.182048 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 10:43:41.182295 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":136,"threads_started":1}
I20260927 10:43:41.184410 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38963
I20260927 10:43:41.184544 1710 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:41.185104 1710 heartbeater.cc:511] Master 127.0.98.62:38963 requested a full tablet report, sending...
I20260927 10:43:41.186803 1506 ts_manager.cc:194] Registered new tserver with Master: c602ab6ddaa948fba950c099a537742e (127.0.98.1:40963)
I20260927 10:43:41.188021 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60269
I20260927 10:43:41.190806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 1579
I20260927 10:43:41.190938 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:43:41.191231 392 external_mini_cluster.cc:1428] Setting key 9941f0a220d7388239f72d75aff34629
I20260927 10:43:41.195597 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.186657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38963 (local address 127.0.98.1:49337)
0927 10:43:41.188386 (+ 1729us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:41.188400 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:41.188497 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:41.188806 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:41.188808 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:41.189017 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:41.189167 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:41.189174 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.189955 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.189958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:41.190711 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:41.190720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.191296 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.191302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:41.191391 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.191947 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:41.191959 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:41.192300 (+ 341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:41.194056 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:41.194060 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:41.194062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:41.194261 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:41.194592 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:41.194595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:41.194597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:41.194684 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:41.195043 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:41.195046 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:41.195143 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 10:43:41.195251 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 10:43:41.195410 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1657,"thread_start_us":106,"threads_started":1}
I20260927 10:43:41.195590 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.186936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49337 (local address 127.0.98.62:38963)
0927 10:43:41.187104 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:41.187112 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:41.188477 (+ 1365us) server_negotiation.cc:407] Connection header received
0927 10:43:41.188576 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:41.188579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:41.188651 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:41.188756 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:41.189294 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.189817 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.190826 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.191166 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.191283 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.192433 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:41.192436 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:41.192439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:41.192466 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:41.193944 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:41.194360 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:41.194365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:41.194367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:41.194495 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:41.194776 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:41.194781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:41.194783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:41.194949 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:41.195054 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:41.195300 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 10:43:41.195412 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 10:43:41.222535 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38963
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:41.336378 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:41.336666 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:41.336712 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:41.340880 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:41.340947 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:41.340988 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:43:41.341068 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:43:41.346407 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:41.347538 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:41.348601 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:41.358390 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:41.358404 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:41.358412 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:41.358575 1719 server_base.cc:1034] running on GCE node
I20260927 10:43:41.359139 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:41.359789 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:41.360980 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790505821360960 us; error 31 us; skew 500 ppm
Sep 27 10:43:41 dist-test-slave-dzhw krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:41.368191 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 10:43:41.369264 1719 webserver.cc:460] Webserver started at http://127.0.98.2:37835/ using document root <none> and password file <none>
I20260927 10:43:41.369861 1719 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:41.369952 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:41.370193 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:41.372459 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "e0dd91c237474b5795f30df0428ade31"
format_stamp: "Formatted at 2026-09-27 10:43:41 on dist-test-slave-dzhw"
server_key: "df7cd1f59a4e201e28afde48088ec55c"
server_key_iv: "da8133e33dc873ba1b8604a3778796a6"
server_key_version: "encryptionkey@0"
I20260927 10:43:41.373059 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e0dd91c237474b5795f30df0428ade31"
format_stamp: "Formatted at 2026-09-27 10:43:41 on dist-test-slave-dzhw"
server_key: "df7cd1f59a4e201e28afde48088ec55c"
server_key_iv: "da8133e33dc873ba1b8604a3778796a6"
server_key_version: "encryptionkey@0"
I20260927 10:43:41.377019 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 10:43:41.379566 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:41.380856 1719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:43:41.381019 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "e0dd91c237474b5795f30df0428ade31"
format_stamp: "Formatted at 2026-09-27 10:43:41 on dist-test-slave-dzhw"
server_key: "df7cd1f59a4e201e28afde48088ec55c"
server_key_iv: "da8133e33dc873ba1b8604a3778796a6"
server_key_version: "encryptionkey@0"
I20260927 10:43:41.381142 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:41.405022 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:41.408397 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:41.408602 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:41.410205 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:41.410279 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:41.410351 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:41.410406 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:43:41 dist-test-slave-dzhw krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:41.421286 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43597
I20260927 10:43:41.421473 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43597 every 8 connection(s)
I20260927 10:43:41.422379 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:43:41.424427 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.411085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33893 (local address 127.0.98.62:38963)
0927 10:43:41.411246 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:41.411252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:41.412099 (+ 847us) server_negotiation.cc:407] Connection header received
0927 10:43:41.413125 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:41.413131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:41.413219 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:41.413279 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:41.414705 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.415341 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.416779 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.417004 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.417113 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.420670 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:41.420674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:41.420676 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:41.420701 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:41.422271 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:41.422931 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:41.422934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:41.422938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:41.423059 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:41.423460 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:41.423463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:41.423464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:41.423673 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:41.423754 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:41.424205 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 10:43:41.424307 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:43:41.425247 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.411494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38963 (local address 127.0.98.2:33893)
0927 10:43:41.411927 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:41.411978 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:41.412878 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:41.413674 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:41.413681 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:41.414016 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:41.414568 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:41.414583 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.415463 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.415466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:41.416660 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:41.416669 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.417162 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.417166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:41.417374 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.418045 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:41.418061 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:41.420469 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:41.422393 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:41.422401 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:41.422440 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:41.422780 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:41.423175 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:41.423178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:41.423180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:41.423345 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:41.423765 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:41.423769 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:41.423999 (+ 230us) client_negotiation.cc:769] Sending connection context
0927 10:43:41.424146 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 10:43:41.424413 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":118,"threads_started":1}
I20260927 10:43:41.426389 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38963
I20260927 10:43:41.426553 1850 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:41.427203 1850 heartbeater.cc:511] Master 127.0.98.62:38963 requested a full tablet report, sending...
I20260927 10:43:41.428318 1507 ts_manager.cc:194] Registered new tserver with Master: e0dd91c237474b5795f30df0428ade31 (127.0.98.2:43597)
I20260927 10:43:41.429118 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33893
I20260927 10:43:41.429445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 1719
I20260927 10:43:41.429558 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:43:41.429857 392 external_mini_cluster.cc:1428] Setting key f556fbdfb0640a340285f46222a4ef76
I20260927 10:43:41.436884 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.428864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38963 (local address 127.0.98.2:56627)
0927 10:43:41.429147 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:41.429159 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:41.429331 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:41.429598 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:41.429603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:41.429820 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:41.429955 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:41.429962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.430883 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.430886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:41.431641 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:41.431649 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.432105 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.432108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:41.432197 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.432758 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:41.432779 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:41.433297 (+ 518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:41.435074 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:41.435078 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:41.435080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:41.435278 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:41.435768 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:41.435771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:41.435773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:41.435872 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:41.436415 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:41.436418 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:41.436520 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 10:43:41.436619 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 10:43:41.436729 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":101,"threads_started":1}
I20260927 10:43:41.437086 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.428944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56627 (local address 127.0.98.62:38963)
0927 10:43:41.429068 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:41.429073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:41.429239 (+ 166us) server_negotiation.cc:407] Connection header received
0927 10:43:41.429419 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:41.429422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:41.429465 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:41.429564 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:41.430063 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.430758 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.431756 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.431995 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.432103 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.433438 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:41.433441 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:41.433443 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:41.433464 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:41.434953 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:41.435553 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:41.435559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:41.435564 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:41.435660 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:41.435973 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:41.435976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:41.435978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:41.436301 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:41.436412 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:41.436712 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 10:43:41.436850 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 10:43:41.457650 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38963
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:41.569341 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:41.569706 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:41.569798 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:41.573902 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:41.574005 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:41.574071 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:43:41.574156 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:43:41.579567 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:41.580687 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:41.581743 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:41.591639 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:41.591648 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:41.591720 1859 server_base.cc:1034] running on GCE node
W20260927 10:43:41.591665 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:41.592296 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:41.592990 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:41.594177 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790505821594162 us; error 28 us; skew 500 ppm
Sep 27 10:43:41 dist-test-slave-dzhw krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:41.597005 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 10:43:41.597968 1859 webserver.cc:460] Webserver started at http://127.0.98.3:36269/ using document root <none> and password file <none>
I20260927 10:43:41.598610 1859 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:41.598703 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:41.598937 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:41.601006 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec"
format_stamp: "Formatted at 2026-09-27 10:43:41 on dist-test-slave-dzhw"
server_key: "544034ed5011db3df65105bf5011ba9b"
server_key_iv: "21f20be79b61d525d61d95fd68c15b90"
server_key_version: "encryptionkey@0"
I20260927 10:43:41.601625 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec"
format_stamp: "Formatted at 2026-09-27 10:43:41 on dist-test-slave-dzhw"
server_key: "544034ed5011db3df65105bf5011ba9b"
server_key_iv: "21f20be79b61d525d61d95fd68c15b90"
server_key_version: "encryptionkey@0"
I20260927 10:43:41.605525 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 10:43:41.608124 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:41.609453 1859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:43:41.609581 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec"
format_stamp: "Formatted at 2026-09-27 10:43:41 on dist-test-slave-dzhw"
server_key: "544034ed5011db3df65105bf5011ba9b"
server_key_iv: "21f20be79b61d525d61d95fd68c15b90"
server_key_version: "encryptionkey@0"
I20260927 10:43:41.609692 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:41.632680 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:41.636175 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:41.636401 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:41.637861 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:41.637939 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:41.638005 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:41.638056 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:43:41 dist-test-slave-dzhw krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:41.649175 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46685
I20260927 10:43:41.649233 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46685 every 8 connection(s)
I20260927 10:43:41.650079 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:43:41.652596 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.638942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54023 (local address 127.0.98.62:38963)
0927 10:43:41.639072 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:41.639076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:41.640099 (+ 1023us) server_negotiation.cc:407] Connection header received
0927 10:43:41.641254 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:41.641257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:41.641310 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:41.641403 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:41.643162 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.644048 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.645465 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.645698 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.645816 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.648561 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:41.648566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:41.648569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:41.648598 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:41.650400 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:41.651047 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:41.651052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:41.651054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:41.651152 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:41.651506 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:41.651509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:41.651511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:41.651706 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:41.651815 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:41.652294 (+ 479us) server_negotiation.cc:299] Negotiation successful
0927 10:43:41.652411 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:43:41.653405 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.639487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38963 (local address 127.0.98.3:54023)
0927 10:43:41.639955 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:41.639996 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:41.641011 (+ 1015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:41.641632 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:41.641644 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:41.642178 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:41.642979 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:41.643005 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.644170 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.644175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:41.645265 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:41.645274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.645889 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.645893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:41.646058 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.646723 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:41.646740 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:41.648369 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:41.650573 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:41.650579 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:41.650597 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:41.650923 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:41.651246 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:41.651251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:41.651252 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:41.651391 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:41.651812 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:41.651819 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:41.652065 (+ 246us) client_negotiation.cc:769] Sending connection context
0927 10:43:41.652252 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 10:43:41.652563 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":123,"threads_started":1}
I20260927 10:43:41.654106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 1859
I20260927 10:43:41.654256 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:43:41.654589 392 external_mini_cluster.cc:1428] Setting key 7e6a1ec77a3bf117dc7b2f957a3b90b1
I20260927 10:43:41.654796 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38963
I20260927 10:43:41.654968 1990 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:41.655553 1990 heartbeater.cc:511] Master 127.0.98.62:38963 requested a full tablet report, sending...
I20260927 10:43:41.656610 1506 ts_manager.cc:194] Registered new tserver with Master: baabe45dfb1a449abd8b2bec0bd3c2ec (127.0.98.3:46685)
I20260927 10:43:41.657392 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54023
I20260927 10:43:41.663769 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.657022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38963 (local address 127.0.98.3:47417)
0927 10:43:41.657308 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:41.657322 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:41.657420 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:41.657672 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:41.657674 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:41.657853 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:41.657981 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:41.657989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.658781 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.658784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:41.659570 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:41.659578 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.659994 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.659997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:41.660059 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.660577 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:41.660598 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:41.660902 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:41.662375 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:41.662377 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:41.662379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:41.662661 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:41.662907 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:41.662909 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:41.662910 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:41.662983 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:41.663312 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:41.663314 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:41.663418 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 10:43:41.663516 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 10:43:41.663632 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":93,"threads_started":1}
I20260927 10:43:41.663865 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.657083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47417 (local address 127.0.98.62:38963)
0927 10:43:41.657195 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:41.657198 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:41.657390 (+ 192us) server_negotiation.cc:407] Connection header received
0927 10:43:41.657509 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:41.657514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:41.657562 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:41.657652 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:41.658092 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.658662 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.659688 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.659896 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.659974 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.661033 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:41.661036 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:41.661038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:41.661068 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:41.662265 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:41.662751 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:41.662754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:41.662755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:41.662824 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:41.663057 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:41.663059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:41.663060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:41.663209 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:41.663281 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:41.663588 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 10:43:41.663695 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:43:41.668146 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 10:43:41 dist-test-slave-dzhw krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505821, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 10:43:41 dist-test-slave-dzhw krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505821, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:41.693926 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.685294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46850 (local address 127.0.98.62:38963)
0927 10:43:41.685481 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:41.685484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:41.685498 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:43:41.685539 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:41.685542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:41.685583 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:41.685708 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:41.686472 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.686987 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.687843 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.688064 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.688174 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.690868 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:41.690871 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:41.690874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:41.690898 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:41.692224 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:41.692710 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:41.692715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:41.692717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:41.692801 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:41.693077 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:41.693082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:41.693084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:41.693264 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:41.693396 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:41.693640 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 10:43:41.693754 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 10:43:41.696987 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:43:41.700183 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:43:41.715391 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.709134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43597 (local address 127.0.0.1:53786)
0927 10:43:41.709877 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:41.709891 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:41.709973 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:41.711009 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:41.711014 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:41.711034 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:43:41.711120 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:41.711126 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.712395 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.712401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:41.713707 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:41.713717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.714826 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.714830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:41.714937 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.714977 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 10:43:41.715111 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 10:43:41.715230 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":210,"thread_start_us":96,"threads_started":1}
I20260927 10:43:41.715432 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.709640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53786 (local address 127.0.98.2:43597)
0927 10:43:41.710527 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:41.710540 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:41.710562 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:43:41.710634 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:41.710639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:41.710849 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:41.711025 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:43:41.711236 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.712285 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.713885 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.714701 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.714814 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.714980 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:41.715143 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 10:43:41.715219 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":111,"threads_started":1}
I20260927 10:43:41.715895 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.708993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40963 (local address 127.0.0.1:41918)
0927 10:43:41.709459 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:41.709506 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:41.709629 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:41.710459 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:41.710465 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:41.710491 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:43:41.710648 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:41.710655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.712216 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.712219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:41.713676 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:41.713687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.715327 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.715330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:41.715415 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.715443 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 10:43:41.715526 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 10:43:41.715600 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":116,"thread_start_us":78,"threads_started":1}
I20260927 10:43:41.716212 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.709660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46685 (local address 127.0.0.1:42534)
0927 10:43:41.710140 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:41.710151 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:41.710212 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:41.710847 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:41.710850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:41.710868 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:43:41.710967 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:41.710977 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.712000 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.712005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:41.713676 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:41.713687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.715795 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.715799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:41.715918 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.715940 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:43:41.716024 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 10:43:41.716091 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":97,"thread_start_us":100,"threads_started":1}
I20260927 10:43:41.716359 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.709300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41918 (local address 127.0.98.1:40963)
0927 10:43:41.709916 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:41.709925 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:41.709952 (+ 27us) server_negotiation.cc:407] Connection header received
0927 10:43:41.710029 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:41.710036 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:41.710256 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:41.710799 (+ 543us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:43:41.710923 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.712103 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.714527 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.715207 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.715309 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.715440 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:41.716141 (+ 701us) server_negotiation.cc:299] Negotiation successful
0927 10:43:41.716231 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":71,"threads_started":1}
I20260927 10:43:41.717322 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.710057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42534 (local address 127.0.98.3:46685)
0927 10:43:41.710481 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:41.710506 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:41.710524 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:43:41.710583 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:41.710587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:41.710717 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:41.710874 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:43:41.711105 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.711879 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.713800 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.714701 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.714814 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.714980 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:41.717089 (+ 2109us) server_negotiation.cc:299] Negotiation successful
0927 10:43:41.717189 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":98,"threads_started":1}
I20260927 10:43:41.718561 1784 tablet_service.cc:1467] Processing CreateTablet for tablet e227a4caff804185bef7ccbcb860f64d (DEFAULT_TABLE table=test-table [id=f7fc3c389a3b403d95a287e5079efd9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:41.718791 1644 tablet_service.cc:1467] Processing CreateTablet for tablet e227a4caff804185bef7ccbcb860f64d (DEFAULT_TABLE table=test-table [id=f7fc3c389a3b403d95a287e5079efd9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:41.719712 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e227a4caff804185bef7ccbcb860f64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:41.719735 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e227a4caff804185bef7ccbcb860f64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:41.719751 1924 tablet_service.cc:1467] Processing CreateTablet for tablet e227a4caff804185bef7ccbcb860f64d (DEFAULT_TABLE table=test-table [id=f7fc3c389a3b403d95a287e5079efd9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:41.720697 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e227a4caff804185bef7ccbcb860f64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:41.725909 2013 tablet_bootstrap.cc:492] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e: Bootstrap starting.
I20260927 10:43:41.728252 2014 tablet_bootstrap.cc:492] T e227a4caff804185bef7ccbcb860f64d P e0dd91c237474b5795f30df0428ade31: Bootstrap starting.
I20260927 10:43:41.728819 2013 tablet_bootstrap.cc:654] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:41.728878 2015 tablet_bootstrap.cc:492] T e227a4caff804185bef7ccbcb860f64d P baabe45dfb1a449abd8b2bec0bd3c2ec: Bootstrap starting.
I20260927 10:43:41.729560 2013 log.cc:824] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e: Log is configured to *not* fsync() on all Append() calls
I20260927 10:43:41.730378 2014 tablet_bootstrap.cc:654] T e227a4caff804185bef7ccbcb860f64d P e0dd91c237474b5795f30df0428ade31: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:41.730792 2015 tablet_bootstrap.cc:654] T e227a4caff804185bef7ccbcb860f64d P baabe45dfb1a449abd8b2bec0bd3c2ec: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:41.731213 2014 log.cc:824] T e227a4caff804185bef7ccbcb860f64d P e0dd91c237474b5795f30df0428ade31: Log is configured to *not* fsync() on all Append() calls
I20260927 10:43:41.731489 2013 tablet_bootstrap.cc:492] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e: No bootstrap required, opened a new log
I20260927 10:43:41.731527 2015 log.cc:824] T e227a4caff804185bef7ccbcb860f64d P baabe45dfb1a449abd8b2bec0bd3c2ec: Log is configured to *not* fsync() on all Append() calls
I20260927 10:43:41.731702 2013 ts_tablet_manager.cc:1397] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:43:41.733352 2015 tablet_bootstrap.cc:492] T e227a4caff804185bef7ccbcb860f64d P baabe45dfb1a449abd8b2bec0bd3c2ec: No bootstrap required, opened a new log
I20260927 10:43:41.733541 2015 ts_tablet_manager.cc:1397] T e227a4caff804185bef7ccbcb860f64d P baabe45dfb1a449abd8b2bec0bd3c2ec: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:43:41.735651 2013 raft_consensus.cc:348] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:41.735896 2013 raft_consensus.cc:374] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:41.735949 2013 raft_consensus.cc:729] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c602ab6ddaa948fba950c099a537742e, State: Initialized, Role: FOLLOWER
I20260927 10:43:41.736099 2014 tablet_bootstrap.cc:492] T e227a4caff804185bef7ccbcb860f64d P e0dd91c237474b5795f30df0428ade31: No bootstrap required, opened a new log
I20260927 10:43:41.736284 2014 ts_tablet_manager.cc:1397] T e227a4caff804185bef7ccbcb860f64d P e0dd91c237474b5795f30df0428ade31: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260927 10:43:41.736483 2013 consensus_queue.cc:260] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:41.737115 1710 heartbeater.cc:503] Master 127.0.98.62:38963 was elected leader, sending a full tablet report...
I20260927 10:43:41.737176 2015 raft_consensus.cc:348] T e227a4caff804185bef7ccbcb860f64d P baabe45dfb1a449abd8b2bec0bd3c2ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:41.737406 2015 raft_consensus.cc:374] T e227a4caff804185bef7ccbcb860f64d P baabe45dfb1a449abd8b2bec0bd3c2ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:41.737457 2015 raft_consensus.cc:729] T e227a4caff804185bef7ccbcb860f64d P baabe45dfb1a449abd8b2bec0bd3c2ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baabe45dfb1a449abd8b2bec0bd3c2ec, State: Initialized, Role: FOLLOWER
I20260927 10:43:41.737464 2013 ts_tablet_manager.cc:1428] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:43:41.737850 2015 consensus_queue.cc:260] T e227a4caff804185bef7ccbcb860f64d P baabe45dfb1a449abd8b2bec0bd3c2ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:41.738741 1990 heartbeater.cc:503] Master 127.0.98.62:38963 was elected leader, sending a full tablet report...
I20260927 10:43:41.738826 2015 ts_tablet_manager.cc:1428] T e227a4caff804185bef7ccbcb860f64d P baabe45dfb1a449abd8b2bec0bd3c2ec: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:43:41.739861 2014 raft_consensus.cc:348] T e227a4caff804185bef7ccbcb860f64d P e0dd91c237474b5795f30df0428ade31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:41.740097 2014 raft_consensus.cc:374] T e227a4caff804185bef7ccbcb860f64d P e0dd91c237474b5795f30df0428ade31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:41.740171 2014 raft_consensus.cc:729] T e227a4caff804185bef7ccbcb860f64d P e0dd91c237474b5795f30df0428ade31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0dd91c237474b5795f30df0428ade31, State: Initialized, Role: FOLLOWER
I20260927 10:43:41.740640 2014 consensus_queue.cc:260] T e227a4caff804185bef7ccbcb860f64d P e0dd91c237474b5795f30df0428ade31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:41.741272 1850 heartbeater.cc:503] Master 127.0.98.62:38963 was elected leader, sending a full tablet report...
I20260927 10:43:41.741597 2014 ts_tablet_manager.cc:1428] T e227a4caff804185bef7ccbcb860f64d P e0dd91c237474b5795f30df0428ade31: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:43:41.746619 2019 raft_consensus.cc:482] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:43:41.746784 2019 raft_consensus.cc:504] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:41.747857 2019 leader_election.cc:288] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0dd91c237474b5795f30df0428ade31 (127.0.98.2:43597), baabe45dfb1a449abd8b2bec0bd3c2ec (127.0.98.3:46685)
I20260927 10:43:41.752835 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.748565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59427 (local address 127.0.98.3:46685)
0927 10:43:41.748692 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:41.748696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:41.748822 (+ 126us) server_negotiation.cc:407] Connection header received
0927 10:43:41.748898 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:41.748902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:41.748950 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:41.749039 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:43:41.749316 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.750055 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.751594 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.752253 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.752393 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.752473 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:41.752632 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 10:43:41.752715 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:43:41.752856 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.748462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46685 (local address 127.0.98.1:59427)
0927 10:43:41.748726 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:41.748746 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:41.748829 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:41.749068 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:41.749071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:41.749083 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:43:41.749203 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:41.749211 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.750166 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.750169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:41.751471 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:41.751482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.752393 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.752397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:41.752497 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.752526 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 10:43:41.752607 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 10:43:41.752686 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":101,"threads_started":1}
I20260927 10:43:41.752918 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.748163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35331 (local address 127.0.98.2:43597)
0927 10:43:41.748279 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:41.748282 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:41.748513 (+ 231us) server_negotiation.cc:407] Connection header received
0927 10:43:41.748711 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:41.748717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:41.748764 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:41.748830 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:43:41.749233 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.750042 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.751757 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.752403 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.752501 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.752551 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:41.752731 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 10:43:41.752793 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:43:41.752868 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.748051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43597 (local address 127.0.98.1:35331)
0927 10:43:41.748409 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:41.748422 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:41.748587 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:41.748973 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:41.748976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:41.748998 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:43:41.749122 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:41.749131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.750505 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.750510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:41.751638 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:41.751645 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.752515 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.752519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:41.752623 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.752641 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 10:43:41.752704 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 10:43:41.752753 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":121,"threads_started":1}
I20260927 10:43:41.753526 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e227a4caff804185bef7ccbcb860f64d" candidate_uuid: "c602ab6ddaa948fba950c099a537742e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" is_pre_election: true
I20260927 10:43:41.753794 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e227a4caff804185bef7ccbcb860f64d" candidate_uuid: "c602ab6ddaa948fba950c099a537742e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0dd91c237474b5795f30df0428ade31" is_pre_election: true
I20260927 10:43:41.753850 1944 raft_consensus.cc:2454] T e227a4caff804185bef7ccbcb860f64d P baabe45dfb1a449abd8b2bec0bd3c2ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c602ab6ddaa948fba950c099a537742e in term 0.
I20260927 10:43:41.754037 1804 raft_consensus.cc:2454] T e227a4caff804185bef7ccbcb860f64d P e0dd91c237474b5795f30df0428ade31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c602ab6ddaa948fba950c099a537742e in term 0.
I20260927 10:43:41.754379 1598 leader_election.cc:302] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: baabe45dfb1a449abd8b2bec0bd3c2ec, c602ab6ddaa948fba950c099a537742e; no voters:
I20260927 10:43:41.754745 2019 raft_consensus.cc:2781] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:43:41.754842 2019 raft_consensus.cc:482] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:43:41.754906 2019 raft_consensus.cc:3037] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:41.756078 2019 raft_consensus.cc:504] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:41.756523 2019 leader_election.cc:288] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e [CANDIDATE]: Term 1 election: Requested vote from peers e0dd91c237474b5795f30df0428ade31 (127.0.98.2:43597), baabe45dfb1a449abd8b2bec0bd3c2ec (127.0.98.3:46685)
I20260927 10:43:41.756949 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e227a4caff804185bef7ccbcb860f64d" candidate_uuid: "c602ab6ddaa948fba950c099a537742e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec"
I20260927 10:43:41.756946 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e227a4caff804185bef7ccbcb860f64d" candidate_uuid: "c602ab6ddaa948fba950c099a537742e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0dd91c237474b5795f30df0428ade31"
I20260927 10:43:41.757095 1944 raft_consensus.cc:3037] T e227a4caff804185bef7ccbcb860f64d P baabe45dfb1a449abd8b2bec0bd3c2ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:41.757119 1804 raft_consensus.cc:3037] T e227a4caff804185bef7ccbcb860f64d P e0dd91c237474b5795f30df0428ade31 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:41.758142 1944 raft_consensus.cc:2454] T e227a4caff804185bef7ccbcb860f64d P baabe45dfb1a449abd8b2bec0bd3c2ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c602ab6ddaa948fba950c099a537742e in term 1.
I20260927 10:43:41.758524 1804 raft_consensus.cc:2454] T e227a4caff804185bef7ccbcb860f64d P e0dd91c237474b5795f30df0428ade31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c602ab6ddaa948fba950c099a537742e in term 1.
I20260927 10:43:41.758574 1598 leader_election.cc:302] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: baabe45dfb1a449abd8b2bec0bd3c2ec, c602ab6ddaa948fba950c099a537742e; no voters:
I20260927 10:43:41.758730 2019 raft_consensus.cc:2781] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:43:41.758984 2019 raft_consensus.cc:686] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e [term 1 LEADER]: Becoming Leader. State: Replica: c602ab6ddaa948fba950c099a537742e, State: Running, Role: LEADER
I20260927 10:43:41.759330 2019 consensus_queue.cc:237] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:41.762693 1507 catalog_manager.cc:5495] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e reported cstate change: term changed from 0 to 1, leader changed from <none> to c602ab6ddaa948fba950c099a537742e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c602ab6ddaa948fba950c099a537742e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } health_report { overall_health: UNKNOWN } } }
I20260927 10:43:41.786109 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.781981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41928 (local address 127.0.98.1:40963)
0927 10:43:41.782103 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:41.782107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:41.782121 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:43:41.782341 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:41.782345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:41.782404 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:41.782534 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:43:41.782761 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.783337 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.784387 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.784652 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.784732 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.785176 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:43:41.785693 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:43:41.785783 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:41.785890 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 10:43:41.785939 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:43:41.792003 1804 raft_consensus.cc:1261] T e227a4caff804185bef7ccbcb860f64d P e0dd91c237474b5795f30df0428ade31 [term 1 FOLLOWER]: Refusing update from remote peer c602ab6ddaa948fba950c099a537742e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:43:41.792186 1944 raft_consensus.cc:1261] T e227a4caff804185bef7ccbcb860f64d P baabe45dfb1a449abd8b2bec0bd3c2ec [term 1 FOLLOWER]: Refusing update from remote peer c602ab6ddaa948fba950c099a537742e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:43:41.792735 2019 consensus_queue.cc:1035] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:43:41.792963 2024 consensus_queue.cc:1035] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e [LEADER]: Connected to new peer: Peer: permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:43:41.811482 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:43:41.811640 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:43:41.814098 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49f0b9e4b209470098b3ce18bd7c1de2: Sending DeleteTablet for 3 replicas of tablet e227a4caff804185bef7ccbcb860f64d
I20260927 10:43:41.814932 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet e227a4caff804185bef7ccbcb860f64d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:43:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53786
I20260927 10:43:41.814929 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet e227a4caff804185bef7ccbcb860f64d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:43:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41918
I20260927 10:43:41.815626 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet e227a4caff804185bef7ccbcb860f64d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:43:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42534
I20260927 10:43:41.816109 2038 tablet_replica.cc:330] T e227a4caff804185bef7ccbcb860f64d P baabe45dfb1a449abd8b2bec0bd3c2ec: stopping tablet replica
I20260927 10:43:41.816239 2037 tablet_replica.cc:330] T e227a4caff804185bef7ccbcb860f64d P e0dd91c237474b5795f30df0428ade31: stopping tablet replica
I20260927 10:43:41.816350 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 10:43:41.816560 2037 raft_consensus.cc:2229] T e227a4caff804185bef7ccbcb860f64d P e0dd91c237474b5795f30df0428ade31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:43:41.816722 2038 raft_consensus.cc:2229] T e227a4caff804185bef7ccbcb860f64d P baabe45dfb1a449abd8b2bec0bd3c2ec [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 10:43:41.816787 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:43:41.816888 2037 raft_consensus.cc:2258] T e227a4caff804185bef7ccbcb860f64d P e0dd91c237474b5795f30df0428ade31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:43:41.817389 2036 tablet_replica.cc:330] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e: stopping tablet replica
I20260927 10:43:41.817672 2036 raft_consensus.cc:2229] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:43:41.818013 2036 raft_consensus.cc:2258] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:43:41.817199 2038 raft_consensus.cc:2258] T e227a4caff804185bef7ccbcb860f64d P baabe45dfb1a449abd8b2bec0bd3c2ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:43:41.818990 2037 ts_tablet_manager.cc:1905] T e227a4caff804185bef7ccbcb860f64d P e0dd91c237474b5795f30df0428ade31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:43:41.819787 2038 ts_tablet_manager.cc:1905] T e227a4caff804185bef7ccbcb860f64d P baabe45dfb1a449abd8b2bec0bd3c2ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:43:41.821810 2037 ts_tablet_manager.cc:1918] T e227a4caff804185bef7ccbcb860f64d P e0dd91c237474b5795f30df0428ade31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:43:41.821978 2037 log.cc:1196] T e227a4caff804185bef7ccbcb860f64d P e0dd91c237474b5795f30df0428ade31: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-1/wal/wals/e227a4caff804185bef7ccbcb860f64d
I20260927 10:43:41.822451 2038 ts_tablet_manager.cc:1918] T e227a4caff804185bef7ccbcb860f64d P baabe45dfb1a449abd8b2bec0bd3c2ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:43:41.822551 2038 log.cc:1196] T e227a4caff804185bef7ccbcb860f64d P baabe45dfb1a449abd8b2bec0bd3c2ec: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/e227a4caff804185bef7ccbcb860f64d
I20260927 10:43:41.822645 1644 tablet_service.cc:1467] Processing CreateTablet for tablet fa169a68b76b4ca0b81285c1abb04a1a (DEFAULT_TABLE table=test-table [id=8f653579e2534374950be011a8c3ab98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:41.822852 2038 ts_tablet_manager.cc:1939] T e227a4caff804185bef7ccbcb860f64d P baabe45dfb1a449abd8b2bec0bd3c2ec: Deleting consensus metadata
I20260927 10:43:41.822916 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa169a68b76b4ca0b81285c1abb04a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:41.822871 1924 tablet_service.cc:1467] Processing CreateTablet for tablet fa169a68b76b4ca0b81285c1abb04a1a (DEFAULT_TABLE table=test-table [id=8f653579e2534374950be011a8c3ab98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:41.823107 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa169a68b76b4ca0b81285c1abb04a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:41.824702 2013 tablet_bootstrap.cc:492] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e: Bootstrap starting.
I20260927 10:43:41.825066 1784 tablet_service.cc:1467] Processing CreateTablet for tablet fa169a68b76b4ca0b81285c1abb04a1a (DEFAULT_TABLE table=test-table [id=8f653579e2534374950be011a8c3ab98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:41.825313 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa169a68b76b4ca0b81285c1abb04a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:41.825697 2013 tablet_bootstrap.cc:654] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:41.826293 2015 tablet_bootstrap.cc:492] T fa169a68b76b4ca0b81285c1abb04a1a P baabe45dfb1a449abd8b2bec0bd3c2ec: Bootstrap starting.
I20260927 10:43:41.827102 2036 ts_tablet_manager.cc:1905] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:43:41.827332 2015 tablet_bootstrap.cc:654] T fa169a68b76b4ca0b81285c1abb04a1a P baabe45dfb1a449abd8b2bec0bd3c2ec: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:41.827836 2014 tablet_bootstrap.cc:492] T fa169a68b76b4ca0b81285c1abb04a1a P e0dd91c237474b5795f30df0428ade31: Bootstrap starting.
I20260927 10:43:41.828706 2014 tablet_bootstrap.cc:654] T fa169a68b76b4ca0b81285c1abb04a1a P e0dd91c237474b5795f30df0428ade31: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:41.829008 1494 catalog_manager.cc:4841] TS baabe45dfb1a449abd8b2bec0bd3c2ec (127.0.98.3:46685): tablet e227a4caff804185bef7ccbcb860f64d (table test-table [id=f7fc3c389a3b403d95a287e5079efd9e]) successfully deleted
I20260927 10:43:41.829924 2015 tablet_bootstrap.cc:492] T fa169a68b76b4ca0b81285c1abb04a1a P baabe45dfb1a449abd8b2bec0bd3c2ec: No bootstrap required, opened a new log
I20260927 10:43:41.830018 2015 ts_tablet_manager.cc:1397] T fa169a68b76b4ca0b81285c1abb04a1a P baabe45dfb1a449abd8b2bec0bd3c2ec: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 10:43:41.830343 2037 ts_tablet_manager.cc:1939] T e227a4caff804185bef7ccbcb860f64d P e0dd91c237474b5795f30df0428ade31: Deleting consensus metadata
I20260927 10:43:41.830549 2015 raft_consensus.cc:348] T fa169a68b76b4ca0b81285c1abb04a1a P baabe45dfb1a449abd8b2bec0bd3c2ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:41.830680 2015 raft_consensus.cc:374] T fa169a68b76b4ca0b81285c1abb04a1a P baabe45dfb1a449abd8b2bec0bd3c2ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:41.830720 2015 raft_consensus.cc:729] T fa169a68b76b4ca0b81285c1abb04a1a P baabe45dfb1a449abd8b2bec0bd3c2ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baabe45dfb1a449abd8b2bec0bd3c2ec, State: Initialized, Role: FOLLOWER
I20260927 10:43:41.830863 2015 consensus_queue.cc:260] T fa169a68b76b4ca0b81285c1abb04a1a P baabe45dfb1a449abd8b2bec0bd3c2ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:41.831146 2015 ts_tablet_manager.cc:1428] T fa169a68b76b4ca0b81285c1abb04a1a P baabe45dfb1a449abd8b2bec0bd3c2ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:43:41.831269 2036 ts_tablet_manager.cc:1918] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:43:41.831354 2036 log.cc:1196] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-0/wal/wals/e227a4caff804185bef7ccbcb860f64d
I20260927 10:43:41.831650 2036 ts_tablet_manager.cc:1939] T e227a4caff804185bef7ccbcb860f64d P c602ab6ddaa948fba950c099a537742e: Deleting consensus metadata
I20260927 10:43:41.832561 1494 catalog_manager.cc:4841] TS e0dd91c237474b5795f30df0428ade31 (127.0.98.2:43597): tablet e227a4caff804185bef7ccbcb860f64d (table test-table [id=f7fc3c389a3b403d95a287e5079efd9e]) successfully deleted
I20260927 10:43:41.833509 1491 catalog_manager.cc:4841] TS c602ab6ddaa948fba950c099a537742e (127.0.98.1:40963): tablet e227a4caff804185bef7ccbcb860f64d (table test-table [id=f7fc3c389a3b403d95a287e5079efd9e]) successfully deleted
I20260927 10:43:41.834103 2014 tablet_bootstrap.cc:492] T fa169a68b76b4ca0b81285c1abb04a1a P e0dd91c237474b5795f30df0428ade31: No bootstrap required, opened a new log
I20260927 10:43:41.834198 2014 ts_tablet_manager.cc:1397] T fa169a68b76b4ca0b81285c1abb04a1a P e0dd91c237474b5795f30df0428ade31: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 10:43:41.834546 2013 tablet_bootstrap.cc:492] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e: No bootstrap required, opened a new log
I20260927 10:43:41.834635 2013 ts_tablet_manager.cc:1397] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260927 10:43:41.834695 2014 raft_consensus.cc:348] T fa169a68b76b4ca0b81285c1abb04a1a P e0dd91c237474b5795f30df0428ade31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:41.834822 2014 raft_consensus.cc:374] T fa169a68b76b4ca0b81285c1abb04a1a P e0dd91c237474b5795f30df0428ade31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:41.834860 2014 raft_consensus.cc:729] T fa169a68b76b4ca0b81285c1abb04a1a P e0dd91c237474b5795f30df0428ade31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0dd91c237474b5795f30df0428ade31, State: Initialized, Role: FOLLOWER
I20260927 10:43:41.834991 2014 consensus_queue.cc:260] T fa169a68b76b4ca0b81285c1abb04a1a P e0dd91c237474b5795f30df0428ade31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:41.835101 2013 raft_consensus.cc:348] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:41.835204 2013 raft_consensus.cc:374] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:41.835244 2013 raft_consensus.cc:729] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c602ab6ddaa948fba950c099a537742e, State: Initialized, Role: FOLLOWER
I20260927 10:43:41.835278 2014 ts_tablet_manager.cc:1428] T fa169a68b76b4ca0b81285c1abb04a1a P e0dd91c237474b5795f30df0428ade31: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:43:41.835386 2013 consensus_queue.cc:260] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:41.835695 2013 ts_tablet_manager.cc:1428] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:43:41.876870 2019 raft_consensus.cc:482] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:43:41.877089 2019 raft_consensus.cc:504] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:41.877545 2019 leader_election.cc:288] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0dd91c237474b5795f30df0428ade31 (127.0.98.2:43597), baabe45dfb1a449abd8b2bec0bd3c2ec (127.0.98.3:46685)
I20260927 10:43:41.877956 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa169a68b76b4ca0b81285c1abb04a1a" candidate_uuid: "c602ab6ddaa948fba950c099a537742e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0dd91c237474b5795f30df0428ade31" is_pre_election: true
I20260927 10:43:41.878000 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa169a68b76b4ca0b81285c1abb04a1a" candidate_uuid: "c602ab6ddaa948fba950c099a537742e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" is_pre_election: true
I20260927 10:43:41.878106 1804 raft_consensus.cc:2454] T fa169a68b76b4ca0b81285c1abb04a1a P e0dd91c237474b5795f30df0428ade31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c602ab6ddaa948fba950c099a537742e in term 0.
I20260927 10:43:41.878118 1944 raft_consensus.cc:2454] T fa169a68b76b4ca0b81285c1abb04a1a P baabe45dfb1a449abd8b2bec0bd3c2ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c602ab6ddaa948fba950c099a537742e in term 0.
I20260927 10:43:41.878463 1598 leader_election.cc:302] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c602ab6ddaa948fba950c099a537742e, e0dd91c237474b5795f30df0428ade31; no voters:
I20260927 10:43:41.878749 2019 raft_consensus.cc:2781] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:43:41.878810 2019 raft_consensus.cc:482] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:43:41.878872 2019 raft_consensus.cc:3037] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:41.879851 2019 raft_consensus.cc:504] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:41.880205 2019 leader_election.cc:288] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e [CANDIDATE]: Term 1 election: Requested vote from peers e0dd91c237474b5795f30df0428ade31 (127.0.98.2:43597), baabe45dfb1a449abd8b2bec0bd3c2ec (127.0.98.3:46685)
I20260927 10:43:41.880512 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa169a68b76b4ca0b81285c1abb04a1a" candidate_uuid: "c602ab6ddaa948fba950c099a537742e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0dd91c237474b5795f30df0428ade31"
I20260927 10:43:41.880537 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa169a68b76b4ca0b81285c1abb04a1a" candidate_uuid: "c602ab6ddaa948fba950c099a537742e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec"
I20260927 10:43:41.880622 1804 raft_consensus.cc:3037] T fa169a68b76b4ca0b81285c1abb04a1a P e0dd91c237474b5795f30df0428ade31 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:41.880628 1944 raft_consensus.cc:3037] T fa169a68b76b4ca0b81285c1abb04a1a P baabe45dfb1a449abd8b2bec0bd3c2ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:41.881686 1944 raft_consensus.cc:2454] T fa169a68b76b4ca0b81285c1abb04a1a P baabe45dfb1a449abd8b2bec0bd3c2ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c602ab6ddaa948fba950c099a537742e in term 1.
I20260927 10:43:41.881682 1804 raft_consensus.cc:2454] T fa169a68b76b4ca0b81285c1abb04a1a P e0dd91c237474b5795f30df0428ade31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c602ab6ddaa948fba950c099a537742e in term 1.
I20260927 10:43:41.881978 1598 leader_election.cc:302] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c602ab6ddaa948fba950c099a537742e, e0dd91c237474b5795f30df0428ade31; no voters:
I20260927 10:43:41.882128 2019 raft_consensus.cc:2781] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:43:41.882225 2019 raft_consensus.cc:686] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e [term 1 LEADER]: Becoming Leader. State: Replica: c602ab6ddaa948fba950c099a537742e, State: Running, Role: LEADER
I20260927 10:43:41.882376 2019 consensus_queue.cc:237] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:41.884235 1507 catalog_manager.cc:5495] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e reported cstate change: term changed from 0 to 1, leader changed from <none> to c602ab6ddaa948fba950c099a537742e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c602ab6ddaa948fba950c099a537742e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } health_report { overall_health: UNKNOWN } } }
W20260927 10:43:41.901165 1991 tablet.cc:2369] T fa169a68b76b4ca0b81285c1abb04a1a P baabe45dfb1a449abd8b2bec0bd3c2ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:43:41.923823 1851 tablet.cc:2369] T fa169a68b76b4ca0b81285c1abb04a1a P e0dd91c237474b5795f30df0428ade31: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 10:43:41 dist-test-slave-dzhw krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260927 10:43:41.932209 1711 tablet.cc:2369] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:43:41.934238 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.924166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38963 (local address 127.0.0.1:46860)
0927 10:43:41.924445 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:41.924460 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:41.924584 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:41.924883 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:41.924885 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:41.925264 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:41.925359 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:41.925365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.926510 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.926512 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:41.927236 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:41.927245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.927867 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.927871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:41.927988 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.928675 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:41.928687 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:41.930276 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:41.932275 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:41.932293 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:41.932302 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:41.932752 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:41.933190 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:41.933193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:41.933195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:41.933291 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:41.933751 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:41.933757 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:41.933864 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 10:43:41.933967 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 10:43:41.934098 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":123,"threads_started":1}
I20260927 10:43:41.934382 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:41.924178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46860 (local address 127.0.98.62:38963)
0927 10:43:41.924305 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:41.924312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:41.924576 (+ 264us) server_negotiation.cc:407] Connection header received
0927 10:43:41.924667 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:41.924671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:41.924724 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:41.924828 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:41.925500 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.926334 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.927358 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:41.927681 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:41.927804 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:41.930456 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:41.930460 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:41.930463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:41.930489 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:41.932068 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:41.932921 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:41.932925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:41.932926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:41.933043 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:41.933399 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:41.933402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:41.933405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:41.933640 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:41.933749 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:41.934064 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 10:43:41.934186 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 10:43:41.938791 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46860:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 10:43:41.939637 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:43:41.944651 1644 tablet_service.cc:1467] Processing CreateTablet for tablet ea64be3417c04ff696dbad6720902d1e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fe05933b46ed431db1fe6992c0afe4d7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:43:41.944731 1784 tablet_service.cc:1467] Processing CreateTablet for tablet ea64be3417c04ff696dbad6720902d1e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fe05933b46ed431db1fe6992c0afe4d7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:43:41.945096 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea64be3417c04ff696dbad6720902d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:41.945098 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea64be3417c04ff696dbad6720902d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:41.947558 2014 tablet_bootstrap.cc:492] T ea64be3417c04ff696dbad6720902d1e P e0dd91c237474b5795f30df0428ade31: Bootstrap starting.
I20260927 10:43:41.947866 2013 tablet_bootstrap.cc:492] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e: Bootstrap starting.
I20260927 10:43:41.948087 1924 tablet_service.cc:1467] Processing CreateTablet for tablet ea64be3417c04ff696dbad6720902d1e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fe05933b46ed431db1fe6992c0afe4d7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:43:41.948457 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea64be3417c04ff696dbad6720902d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:41.948523 2014 tablet_bootstrap.cc:654] T ea64be3417c04ff696dbad6720902d1e P e0dd91c237474b5795f30df0428ade31: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:41.949270 2013 tablet_bootstrap.cc:654] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:41.950644 2013 tablet_bootstrap.cc:492] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e: No bootstrap required, opened a new log
I20260927 10:43:41.950738 2013 ts_tablet_manager.cc:1397] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:43:41.951184 2015 tablet_bootstrap.cc:492] T ea64be3417c04ff696dbad6720902d1e P baabe45dfb1a449abd8b2bec0bd3c2ec: Bootstrap starting.
I20260927 10:43:41.951174 2013 raft_consensus.cc:348] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:41.951305 2013 raft_consensus.cc:374] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:41.951339 2013 raft_consensus.cc:729] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c602ab6ddaa948fba950c099a537742e, State: Initialized, Role: FOLLOWER
I20260927 10:43:41.951458 2013 consensus_queue.cc:260] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:41.951732 2019 tablet_replica.cc:439] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } } }
I20260927 10:43:41.951848 2019 tablet_replica.cc:442] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:43:41.952173 2015 tablet_bootstrap.cc:654] T ea64be3417c04ff696dbad6720902d1e P baabe45dfb1a449abd8b2bec0bd3c2ec: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:41.952922 2013 ts_tablet_manager.cc:1428] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:43:41.953120 2014 tablet_bootstrap.cc:492] T ea64be3417c04ff696dbad6720902d1e P e0dd91c237474b5795f30df0428ade31: No bootstrap required, opened a new log
I20260927 10:43:41.953207 2014 ts_tablet_manager.cc:1397] T ea64be3417c04ff696dbad6720902d1e P e0dd91c237474b5795f30df0428ade31: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 10:43:41.953702 2014 raft_consensus.cc:348] T ea64be3417c04ff696dbad6720902d1e P e0dd91c237474b5795f30df0428ade31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:41.953809 2014 raft_consensus.cc:374] T ea64be3417c04ff696dbad6720902d1e P e0dd91c237474b5795f30df0428ade31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:41.953831 2014 raft_consensus.cc:729] T ea64be3417c04ff696dbad6720902d1e P e0dd91c237474b5795f30df0428ade31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0dd91c237474b5795f30df0428ade31, State: Initialized, Role: FOLLOWER
I20260927 10:43:41.953975 2014 consensus_queue.cc:260] T ea64be3417c04ff696dbad6720902d1e P e0dd91c237474b5795f30df0428ade31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:41.954279 2014 ts_tablet_manager.cc:1428] T ea64be3417c04ff696dbad6720902d1e P e0dd91c237474b5795f30df0428ade31: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:43:41.954523 2021 tablet_replica.cc:439] T ea64be3417c04ff696dbad6720902d1e P e0dd91c237474b5795f30df0428ade31: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } } }
I20260927 10:43:41.954669 2021 tablet_replica.cc:442] T ea64be3417c04ff696dbad6720902d1e P e0dd91c237474b5795f30df0428ade31: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:43:41.955546 2015 tablet_bootstrap.cc:492] T ea64be3417c04ff696dbad6720902d1e P baabe45dfb1a449abd8b2bec0bd3c2ec: No bootstrap required, opened a new log
I20260927 10:43:41.955652 2015 ts_tablet_manager.cc:1397] T ea64be3417c04ff696dbad6720902d1e P baabe45dfb1a449abd8b2bec0bd3c2ec: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 10:43:41.956115 2015 raft_consensus.cc:348] T ea64be3417c04ff696dbad6720902d1e P baabe45dfb1a449abd8b2bec0bd3c2ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:41.956226 2015 raft_consensus.cc:374] T ea64be3417c04ff696dbad6720902d1e P baabe45dfb1a449abd8b2bec0bd3c2ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:41.956265 2015 raft_consensus.cc:729] T ea64be3417c04ff696dbad6720902d1e P baabe45dfb1a449abd8b2bec0bd3c2ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baabe45dfb1a449abd8b2bec0bd3c2ec, State: Initialized, Role: FOLLOWER
I20260927 10:43:41.956395 2015 consensus_queue.cc:260] T ea64be3417c04ff696dbad6720902d1e P baabe45dfb1a449abd8b2bec0bd3c2ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:41.956696 2015 ts_tablet_manager.cc:1428] T ea64be3417c04ff696dbad6720902d1e P baabe45dfb1a449abd8b2bec0bd3c2ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:43:41.956651 2020 tablet_replica.cc:439] T ea64be3417c04ff696dbad6720902d1e P baabe45dfb1a449abd8b2bec0bd3c2ec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } } }
I20260927 10:43:41.956786 2020 tablet_replica.cc:442] T ea64be3417c04ff696dbad6720902d1e P baabe45dfb1a449abd8b2bec0bd3c2ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:43:42.001510 2019 raft_consensus.cc:482] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:43:42.001641 2019 raft_consensus.cc:504] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:42.002151 2019 leader_election.cc:288] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0dd91c237474b5795f30df0428ade31 (127.0.98.2:43597), baabe45dfb1a449abd8b2bec0bd3c2ec (127.0.98.3:46685)
I20260927 10:43:42.002578 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea64be3417c04ff696dbad6720902d1e" candidate_uuid: "c602ab6ddaa948fba950c099a537742e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" is_pre_election: true
I20260927 10:43:42.002625 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea64be3417c04ff696dbad6720902d1e" candidate_uuid: "c602ab6ddaa948fba950c099a537742e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0dd91c237474b5795f30df0428ade31" is_pre_election: true
I20260927 10:43:42.002738 1944 raft_consensus.cc:2454] T ea64be3417c04ff696dbad6720902d1e P baabe45dfb1a449abd8b2bec0bd3c2ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c602ab6ddaa948fba950c099a537742e in term 0.
I20260927 10:43:42.002748 1804 raft_consensus.cc:2454] T ea64be3417c04ff696dbad6720902d1e P e0dd91c237474b5795f30df0428ade31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c602ab6ddaa948fba950c099a537742e in term 0.
I20260927 10:43:42.003007 1598 leader_election.cc:302] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: baabe45dfb1a449abd8b2bec0bd3c2ec, c602ab6ddaa948fba950c099a537742e; no voters:
I20260927 10:43:42.003206 2019 raft_consensus.cc:2781] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:43:42.003285 2019 raft_consensus.cc:482] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:43:42.003338 2019 raft_consensus.cc:3037] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:42.004283 2019 raft_consensus.cc:504] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:42.004630 2019 leader_election.cc:288] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e [CANDIDATE]: Term 1 election: Requested vote from peers e0dd91c237474b5795f30df0428ade31 (127.0.98.2:43597), baabe45dfb1a449abd8b2bec0bd3c2ec (127.0.98.3:46685)
I20260927 10:43:42.005213 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea64be3417c04ff696dbad6720902d1e" candidate_uuid: "c602ab6ddaa948fba950c099a537742e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec"
I20260927 10:43:42.005203 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea64be3417c04ff696dbad6720902d1e" candidate_uuid: "c602ab6ddaa948fba950c099a537742e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0dd91c237474b5795f30df0428ade31"
I20260927 10:43:42.005333 1804 raft_consensus.cc:3037] T ea64be3417c04ff696dbad6720902d1e P e0dd91c237474b5795f30df0428ade31 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:42.005332 1944 raft_consensus.cc:3037] T ea64be3417c04ff696dbad6720902d1e P baabe45dfb1a449abd8b2bec0bd3c2ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:42.006398 1944 raft_consensus.cc:2454] T ea64be3417c04ff696dbad6720902d1e P baabe45dfb1a449abd8b2bec0bd3c2ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c602ab6ddaa948fba950c099a537742e in term 1.
I20260927 10:43:42.006398 1804 raft_consensus.cc:2454] T ea64be3417c04ff696dbad6720902d1e P e0dd91c237474b5795f30df0428ade31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c602ab6ddaa948fba950c099a537742e in term 1.
I20260927 10:43:42.006706 1598 leader_election.cc:302] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: baabe45dfb1a449abd8b2bec0bd3c2ec, c602ab6ddaa948fba950c099a537742e; no voters:
I20260927 10:43:42.006863 2019 raft_consensus.cc:2781] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:43:42.006958 2019 raft_consensus.cc:686] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e [term 1 LEADER]: Becoming Leader. State: Replica: c602ab6ddaa948fba950c099a537742e, State: Running, Role: LEADER
I20260927 10:43:42.007090 2019 consensus_queue.cc:237] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } }
I20260927 10:43:42.007776 2019 tablet_replica.cc:439] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e: TxnStatusTablet state changed. Reason: New leader c602ab6ddaa948fba950c099a537742e. Latest consensus state: current_term: 1 leader_uuid: "c602ab6ddaa948fba950c099a537742e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } } }
I20260927 10:43:42.007875 2019 tablet_replica.cc:442] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:43:42.008157 2053 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 10:43:42.008386 2053 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 10:43:42.008996 1506 catalog_manager.cc:5495] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e reported cstate change: term changed from 0 to 1, leader changed from <none> to c602ab6ddaa948fba950c099a537742e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c602ab6ddaa948fba950c099a537742e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } health_report { overall_health: UNKNOWN } } }
I20260927 10:43:42.023816 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:42.020399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40963 (local address 127.0.0.1:41944)
0927 10:43:42.020525 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:42.020541 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:42.020641 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:42.020913 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:42.020915 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:42.020922 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:43:42.021013 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:42.021018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:42.021986 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:42.021989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:42.022658 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:42.022667 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:42.023205 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:42.023208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:42.023266 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:42.023306 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:43:42.023581 (+ 275us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:43:42.023591 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 10:43:42.023640 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 10:43:42.023704 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 10:43:42.023841 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:42.020498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41944 (local address 127.0.98.1:40963)
0927 10:43:42.020619 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:42.020623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:42.020639 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:43:42.020756 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:42.020759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:42.020811 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:42.020893 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:43:42.021106 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:42.021878 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:42.022768 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:42.023100 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:42.023193 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:42.023418 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:43:42.023504 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:43:42.023572 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:42.023669 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 10:43:42.023714 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:43:42.026209 1804 raft_consensus.cc:1261] T ea64be3417c04ff696dbad6720902d1e P e0dd91c237474b5795f30df0428ade31 [term 1 FOLLOWER]: Refusing update from remote peer c602ab6ddaa948fba950c099a537742e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:43:42.026321 1944 raft_consensus.cc:1261] T ea64be3417c04ff696dbad6720902d1e P baabe45dfb1a449abd8b2bec0bd3c2ec [term 1 FOLLOWER]: Refusing update from remote peer c602ab6ddaa948fba950c099a537742e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:43:42.026757 2019 consensus_queue.cc:1035] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e [LEADER]: Connected to new peer: Peer: permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:43:42.026885 2024 consensus_queue.cc:1035] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:43:42.028921 2020 tablet_replica.cc:439] T ea64be3417c04ff696dbad6720902d1e P baabe45dfb1a449abd8b2bec0bd3c2ec: TxnStatusTablet state changed. Reason: New leader c602ab6ddaa948fba950c099a537742e. Latest consensus state: current_term: 1 leader_uuid: "c602ab6ddaa948fba950c099a537742e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } } }
I20260927 10:43:42.029060 2020 tablet_replica.cc:442] T ea64be3417c04ff696dbad6720902d1e P baabe45dfb1a449abd8b2bec0bd3c2ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:43:42.029352 2021 tablet_replica.cc:439] T ea64be3417c04ff696dbad6720902d1e P e0dd91c237474b5795f30df0428ade31: TxnStatusTablet state changed. Reason: New leader c602ab6ddaa948fba950c099a537742e. Latest consensus state: current_term: 1 leader_uuid: "c602ab6ddaa948fba950c099a537742e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } } }
I20260927 10:43:42.029474 2021 tablet_replica.cc:442] T ea64be3417c04ff696dbad6720902d1e P e0dd91c237474b5795f30df0428ade31: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:43:42.030360 2019 tablet_replica.cc:439] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c602ab6ddaa948fba950c099a537742e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } } }
I20260927 10:43:42.030507 2019 tablet_replica.cc:442] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:43:42.031802 2024 tablet_replica.cc:439] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c602ab6ddaa948fba950c099a537742e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } } }
I20260927 10:43:42.031905 2024 tablet_replica.cc:442] T ea64be3417c04ff696dbad6720902d1e P c602ab6ddaa948fba950c099a537742e: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:43:42.032523 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333911847015280640 to 7333911846942380032. Current Snapshot: MvccSnapshot[applied={T|T < 7333911847015280640}]
I20260927 10:43:42.032702 2021 tablet_replica.cc:439] T ea64be3417c04ff696dbad6720902d1e P e0dd91c237474b5795f30df0428ade31: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c602ab6ddaa948fba950c099a537742e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } } }
I20260927 10:43:42.032792 2021 tablet_replica.cc:442] T ea64be3417c04ff696dbad6720902d1e P e0dd91c237474b5795f30df0428ade31: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:43:42.032961 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333911847015280640 to 7333911846942380032. Current Snapshot: MvccSnapshot[applied={T|T < 7333911847015280640 or (T in {7333911847015280640})}]
I20260927 10:43:42.033694 2020 tablet_replica.cc:439] T ea64be3417c04ff696dbad6720902d1e P baabe45dfb1a449abd8b2bec0bd3c2ec: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c602ab6ddaa948fba950c099a537742e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c602ab6ddaa948fba950c099a537742e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40963 } } peers { permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 } } peers { permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 } } }
I20260927 10:43:42.033794 2020 tablet_replica.cc:442] T ea64be3417c04ff696dbad6720902d1e P baabe45dfb1a449abd8b2bec0bd3c2ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:43:42.046921 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:42.043342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52317 (local address 127.0.98.1:40963)
0927 10:43:42.043481 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:42.043486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:42.043716 (+ 230us) server_negotiation.cc:407] Connection header received
0927 10:43:42.043837 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:42.043839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:42.043879 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:42.043943 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:43:42.044186 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:42.044681 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:42.045870 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:42.046072 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:42.046141 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:42.046487 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:43:42.046566 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:43:42.046622 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:42.046749 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 10:43:42.046793 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:43:42.046964 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:42.043243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40963 (local address 127.0.98.1:52317)
0927 10:43:42.043626 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:42.043640 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:42.043746 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:42.043977 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:42.043980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:42.043987 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:43:42.044090 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:42.044096 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:42.044782 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:42.044786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:42.045749 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:42.045760 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:42.046174 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:42.046178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:42.046261 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:42.046311 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:43:42.046653 (+ 342us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:43:42.046659 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:43:42.046727 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 10:43:42.046808 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":107,"threads_started":1}
I20260927 10:43:42.053853 1944 raft_consensus.cc:1261] T fa169a68b76b4ca0b81285c1abb04a1a P baabe45dfb1a449abd8b2bec0bd3c2ec [term 1 FOLLOWER]: Refusing update from remote peer c602ab6ddaa948fba950c099a537742e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:43:42.053834 1804 raft_consensus.cc:1261] T fa169a68b76b4ca0b81285c1abb04a1a P e0dd91c237474b5795f30df0428ade31 [term 1 FOLLOWER]: Refusing update from remote peer c602ab6ddaa948fba950c099a537742e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:43:42.054214 2024 consensus_queue.cc:1035] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e [LEADER]: Connected to new peer: Peer: permanent_uuid: "baabe45dfb1a449abd8b2bec0bd3c2ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:43:42.054399 2060 consensus_queue.cc:1035] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0dd91c237474b5795f30df0428ade31" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:43:42.058568 2029 mvcc.cc:205] Tried to move back new op lower bound from 7333911847128776704 to 7333911846432145408. Current Snapshot: MvccSnapshot[applied={T|T < 7333911847128776704}]
I20260927 10:43:42.089792 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:43:42.089964 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:43:42.091964 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49f0b9e4b209470098b3ce18bd7c1de2: Sending DeleteTablet for 3 replicas of tablet fa169a68b76b4ca0b81285c1abb04a1a
I20260927 10:43:42.092470 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet fa169a68b76b4ca0b81285c1abb04a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:43:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41918
I20260927 10:43:42.092630 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet fa169a68b76b4ca0b81285c1abb04a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:43:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42534
I20260927 10:43:42.092644 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet fa169a68b76b4ca0b81285c1abb04a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:43:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53786
I20260927 10:43:42.092792 2038 tablet_replica.cc:330] T fa169a68b76b4ca0b81285c1abb04a1a P baabe45dfb1a449abd8b2bec0bd3c2ec: stopping tablet replica
I20260927 10:43:42.092847 2037 tablet_replica.cc:330] T fa169a68b76b4ca0b81285c1abb04a1a P e0dd91c237474b5795f30df0428ade31: stopping tablet replica
I20260927 10:43:42.092855 2036 tablet_replica.cc:330] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e: stopping tablet replica
I20260927 10:43:42.092903 2038 raft_consensus.cc:2229] T fa169a68b76b4ca0b81285c1abb04a1a P baabe45dfb1a449abd8b2bec0bd3c2ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:43:42.092948 2037 raft_consensus.cc:2229] T fa169a68b76b4ca0b81285c1abb04a1a P e0dd91c237474b5795f30df0428ade31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:43:42.092983 2036 raft_consensus.cc:2229] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:43:42.093014 2038 raft_consensus.cc:2258] T fa169a68b76b4ca0b81285c1abb04a1a P baabe45dfb1a449abd8b2bec0bd3c2ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:43:42.093044 2037 raft_consensus.cc:2258] T fa169a68b76b4ca0b81285c1abb04a1a P e0dd91c237474b5795f30df0428ade31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:43:42.093163 2036 raft_consensus.cc:2258] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:43:42.093999 2037 ts_tablet_manager.cc:1905] T fa169a68b76b4ca0b81285c1abb04a1a P e0dd91c237474b5795f30df0428ade31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:43:42.094825 2036 ts_tablet_manager.cc:1905] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:43:42.096830 2037 ts_tablet_manager.cc:1918] T fa169a68b76b4ca0b81285c1abb04a1a P e0dd91c237474b5795f30df0428ade31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:43:42.096912 2037 log.cc:1196] T fa169a68b76b4ca0b81285c1abb04a1a P e0dd91c237474b5795f30df0428ade31: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-1/wal/wals/fa169a68b76b4ca0b81285c1abb04a1a
I20260927 10:43:42.096882 2036 ts_tablet_manager.cc:1918] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:43:42.096987 2036 log.cc:1196] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-0/wal/wals/fa169a68b76b4ca0b81285c1abb04a1a
I20260927 10:43:42.097185 2037 ts_tablet_manager.cc:1939] T fa169a68b76b4ca0b81285c1abb04a1a P e0dd91c237474b5795f30df0428ade31: Deleting consensus metadata
I20260927 10:43:42.097247 2036 ts_tablet_manager.cc:1939] T fa169a68b76b4ca0b81285c1abb04a1a P c602ab6ddaa948fba950c099a537742e: Deleting consensus metadata
I20260927 10:43:42.097896 1494 catalog_manager.cc:4841] TS e0dd91c237474b5795f30df0428ade31 (127.0.98.2:43597): tablet fa169a68b76b4ca0b81285c1abb04a1a (table test-table [id=8f653579e2534374950be011a8c3ab98]) successfully deleted
I20260927 10:43:42.097990 1491 catalog_manager.cc:4841] TS c602ab6ddaa948fba950c099a537742e (127.0.98.1:40963): tablet fa169a68b76b4ca0b81285c1abb04a1a (table test-table [id=8f653579e2534374950be011a8c3ab98]) successfully deleted
I20260927 10:43:42.098502 2038 ts_tablet_manager.cc:1905] T fa169a68b76b4ca0b81285c1abb04a1a P baabe45dfb1a449abd8b2bec0bd3c2ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:43:42.100808 2038 ts_tablet_manager.cc:1918] T fa169a68b76b4ca0b81285c1abb04a1a P baabe45dfb1a449abd8b2bec0bd3c2ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:43:42.100906 2038 log.cc:1196] T fa169a68b76b4ca0b81285c1abb04a1a P baabe45dfb1a449abd8b2bec0bd3c2ec: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/fa169a68b76b4ca0b81285c1abb04a1a
I20260927 10:43:42.101137 2038 ts_tablet_manager.cc:1939] T fa169a68b76b4ca0b81285c1abb04a1a P baabe45dfb1a449abd8b2bec0bd3c2ec: Deleting consensus metadata
I20260927 10:43:42.101913 1494 catalog_manager.cc:4841] TS baabe45dfb1a449abd8b2bec0bd3c2ec (127.0.98.3:46685): tablet fa169a68b76b4ca0b81285c1abb04a1a (table test-table [id=8f653579e2534374950be011a8c3ab98]) successfully deleted
Sep 27 10:43:42 dist-test-slave-dzhw krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505821, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 10:43:42.110491 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:42.099366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41946 (local address 127.0.98.1:40963)
0927 10:43:42.099482 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:42.099486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:42.099713 (+ 227us) server_negotiation.cc:407] Connection header received
0927 10:43:42.099885 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:42.099888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:42.099933 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:42.099983 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:42.100925 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:42.101728 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:42.102947 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:42.103169 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:42.103296 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:42.106747 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:42.106757 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:42.106771 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:42.106796 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:42.108614 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:42.109073 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:42.109079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:42.109080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:42.109156 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:42.109443 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:42.109447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:42.109449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:42.109778 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:42.109999 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:42.110182 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 10:43:42.110307 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 10:43:42.110952 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41946
I20260927 10:43:42.120122 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:42.112659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46874 (local address 127.0.98.62:38963)
0927 10:43:42.112804 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:42.112809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:42.112826 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:43:42.112956 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:42.112959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:42.113006 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:42.113087 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:42.113705 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:42.114571 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:42.115318 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:42.115663 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:42.115756 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:42.117067 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:42.117071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:42.117074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:42.117097 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:42.118613 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:42.118979 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:42.118982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:42.118984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:42.119064 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:42.119299 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:42.119302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:42.119304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:42.119519 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:42.119646 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:42.119847 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 10:43:42.119958 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 10:43:42.120507 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46874
I20260927 10:43:42.121757 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 1579
I20260927 10:43:42.129248 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 1719
I20260927 10:43:42.135123 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 1859
I20260927 10:43:42.140827 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 1475
2026-09-27T10:43:42Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2623 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:43:42 dist-test-slave-dzhw krb5kdc[2070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:43:42 dist-test-slave-dzhw krb5kdc[2070](info): set up 2 sockets
Sep 27 10:43:42 dist-test-slave-dzhw krb5kdc[2070](info): commencing operation
krb5kdc: starting...
W20260927 10:43:43.190387 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:43:43 dist-test-slave-dzhw krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:43:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:43:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:43:43.313596 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36115
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:43.424969 2083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:43.425254 2083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:43.425294 2083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:43.429373 2083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:43:43.429440 2083 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 10:43:43.429456 2083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:43.429514 2083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:43.429548 2083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:43:43.429569 2083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:43:43.435079 2083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36115
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:43.436201 2083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:43.437261 2083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:43:43.446131 2083 server_base.cc:1034] running on GCE node
W20260927 10:43:43.446079 2089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:43.446075 2091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:43.446075 2088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:43.446941 2083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:43.448089 2083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:43.449290 2083 hybrid_clock.cc:648] HybridClock initialized: now 1790505823449258 us; error 47 us; skew 500 ppm
Sep 27 10:43:43 dist-test-slave-dzhw krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:43.452091 2083 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:43:43.453076 2083 webserver.cc:460] Webserver started at http://127.0.98.62:44173/ using document root <none> and password file <none>
I20260927 10:43:43.453673 2083 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:43.453765 2083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:43.453990 2083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:43.456161 2083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "fe9200f9a065422fa6ad6eccded6f5d2"
format_stamp: "Formatted at 2026-09-27 10:43:43 on dist-test-slave-dzhw"
server_key: "e116c78226f606b530549c04d50713ba"
server_key_iv: "97ccacac1cca8199eca5a04cb07f9d3e"
server_key_version: "encryptionkey@0"
I20260927 10:43:43.456737 2083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "fe9200f9a065422fa6ad6eccded6f5d2"
format_stamp: "Formatted at 2026-09-27 10:43:43 on dist-test-slave-dzhw"
server_key: "e116c78226f606b530549c04d50713ba"
server_key_iv: "97ccacac1cca8199eca5a04cb07f9d3e"
server_key_version: "encryptionkey@0"
I20260927 10:43:43.460839 2083 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260927 10:43:43.464015 2098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:43.465322 2083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 10:43:43.465461 2083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "fe9200f9a065422fa6ad6eccded6f5d2"
format_stamp: "Formatted at 2026-09-27 10:43:43 on dist-test-slave-dzhw"
server_key: "e116c78226f606b530549c04d50713ba"
server_key_iv: "97ccacac1cca8199eca5a04cb07f9d3e"
server_key_version: "encryptionkey@0"
I20260927 10:43:43.465584 2083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:43.492544 2083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:43.495740 2083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:43.495954 2083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:43.504043 2083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36115
I20260927 10:43:43.504060 2160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36115 every 8 connection(s)
I20260927 10:43:43.505012 2083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:43:43.507843 2161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:43.510263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 2083
I20260927 10:43:43.510356 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:43:43.510654 392 external_mini_cluster.cc:1428] Setting key cb3ceda80cdc2c9f1a7eb62eff2d3990
I20260927 10:43:43.514240 2161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2: Bootstrap starting.
I20260927 10:43:43.517036 2161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:43.517918 2161 log.cc:824] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:43:43 dist-test-slave-dzhw krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:43.520394 2161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2: No bootstrap required, opened a new log
I20260927 10:43:43.524121 2161 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9200f9a065422fa6ad6eccded6f5d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } }
I20260927 10:43:43.524406 2161 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:43.524472 2161 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe9200f9a065422fa6ad6eccded6f5d2, State: Initialized, Role: FOLLOWER
I20260927 10:43:43.524979 2161 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9200f9a065422fa6ad6eccded6f5d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } }
I20260927 10:43:43.525147 2161 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:43:43.525228 2161 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:43:43.525354 2161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:43.525626 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:43.512005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56096 (local address 127.0.98.62:36115)
0927 10:43:43.512428 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:43.512448 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:43.512484 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:43:43.513128 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:43.513145 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:43.513505 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:43.513792 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:43.514728 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:43.515779 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:43.516956 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:43.517572 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:43.517706 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:43.520523 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:43.520529 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:43.520549 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:43.520586 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:43.522488 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:43.522980 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:43.522986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:43.522991 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:43.523104 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:43.523408 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:43.523411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:43.523415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:43.523837 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:43.524050 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:43.524350 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 10:43:43.524665 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":143,"threads_started":1}
I20260927 10:43:43.526393 2161 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9200f9a065422fa6ad6eccded6f5d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } }
I20260927 10:43:43.526799 2161 leader_election.cc:302] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe9200f9a065422fa6ad6eccded6f5d2; no voters:
I20260927 10:43:43.527161 2161 leader_election.cc:288] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:43:43.527247 2166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:43:43.527459 2166 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2 [term 1 LEADER]: Becoming Leader. State: Replica: fe9200f9a065422fa6ad6eccded6f5d2, State: Running, Role: LEADER
I20260927 10:43:43.527758 2166 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9200f9a065422fa6ad6eccded6f5d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } }
I20260927 10:43:43.528493 2161 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:43:43.528954 2166 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe9200f9a065422fa6ad6eccded6f5d2. Latest consensus state: current_term: 1 leader_uuid: "fe9200f9a065422fa6ad6eccded6f5d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9200f9a065422fa6ad6eccded6f5d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } } }
I20260927 10:43:43.529068 2166 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2 [sys.catalog]: This master's current role is: LEADER
I20260927 10:43:43.529466 2170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:43:43.529445 2167 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe9200f9a065422fa6ad6eccded6f5d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9200f9a065422fa6ad6eccded6f5d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36115 } } }
I20260927 10:43:43.529547 2167 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2 [sys.catalog]: This master's current role is: LEADER
I20260927 10:43:43.532887 2170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:43:43.538267 2170 catalog_manager.cc:1347] Generated new cluster ID: 90b9f755d28f4b84ae323a51c81a6d94
I20260927 10:43:43.538342 2170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:43:43.571060 2170 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:43:43.572316 2170 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:43:43.579712 2170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2: Generated new TSK 0
I20260927 10:43:43.580574 2170 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 10:43:43.617982 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36115
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:43.730990 2187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:43.731258 2187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:43.731305 2187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:43.735611 2187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:43.735689 2187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:43.735730 2187 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 10:43:43.735744 2187 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:43:43.735777 2187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:43:43.741737 2187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:43.742857 2187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:43.743940 2187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:43:43.754010 2187 server_base.cc:1034] running on GCE node
W20260927 10:43:43.753919 2195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:43.753921 2192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:43.753957 2193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:43.754660 2187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:43.755331 2187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:43.756524 2187 hybrid_clock.cc:648] HybridClock initialized: now 1790505823756509 us; error 29 us; skew 500 ppm
Sep 27 10:43:43 dist-test-slave-dzhw krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:43.759409 2187 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 10:43:43.760282 2187 webserver.cc:460] Webserver started at http://127.0.98.1:33125/ using document root <none> and password file <none>
I20260927 10:43:43.760846 2187 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:43.760946 2187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:43.761178 2187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:43.763304 2187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "16badd963c564d45ad782203936af1ae"
format_stamp: "Formatted at 2026-09-27 10:43:43 on dist-test-slave-dzhw"
server_key: "c372572e9c39d386c80cdb05d2f1a0ef"
server_key_iv: "cb6bff731b3f564a022d6c6c1cc08a2d"
server_key_version: "encryptionkey@0"
I20260927 10:43:43.763867 2187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "16badd963c564d45ad782203936af1ae"
format_stamp: "Formatted at 2026-09-27 10:43:43 on dist-test-slave-dzhw"
server_key: "c372572e9c39d386c80cdb05d2f1a0ef"
server_key_iv: "cb6bff731b3f564a022d6c6c1cc08a2d"
server_key_version: "encryptionkey@0"
I20260927 10:43:43.767618 2187 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 10:43:43.770542 2202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:43.771924 2187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:43:43.772089 2187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "16badd963c564d45ad782203936af1ae"
format_stamp: "Formatted at 2026-09-27 10:43:43 on dist-test-slave-dzhw"
server_key: "c372572e9c39d386c80cdb05d2f1a0ef"
server_key_iv: "cb6bff731b3f564a022d6c6c1cc08a2d"
server_key_version: "encryptionkey@0"
I20260927 10:43:43.772238 2187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:43.790340 2187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:43.794113 2187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:43.794348 2187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:43.795897 2187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:43.795986 2187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:43.796066 2187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:43.796116 2187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:43:43 dist-test-slave-dzhw krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:43.807885 2187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44899
I20260927 10:43:43.807964 2317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44899 every 8 connection(s)
I20260927 10:43:43.809073 2187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:43:43.811539 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:43.796868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51461 (local address 127.0.98.62:36115)
0927 10:43:43.796994 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:43.796997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:43.798147 (+ 1150us) server_negotiation.cc:407] Connection header received
0927 10:43:43.799194 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:43.799199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:43.799253 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:43.799313 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:43.801205 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:43.802233 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:43.804141 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:43.804549 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:43.804688 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:43.807788 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:43.807792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:43.807797 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:43.807853 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:43.809441 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:43.810088 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:43.810095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:43.810096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:43.810194 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:43.810578 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:43.810580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:43.810582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:43.810765 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:43.810851 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:43.811305 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 10:43:43.811420 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:43:43.812340 2211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:43.797486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36115 (local address 127.0.98.1:51461)
0927 10:43:43.797997 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:43.798041 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:43.798941 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:43.799769 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:43.799779 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:43.800274 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:43.801014 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:43.801037 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:43.802407 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:43.802436 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:43.803989 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:43.804002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:43.804750 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:43.804753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:43.804903 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:43.805624 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:43.805644 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:43.807607 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:43.809570 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:43.809576 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:43.809592 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:43.809971 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:43.810295 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:43.810298 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:43.810299 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:43.810487 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:43.810853 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:43.810858 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:43.811096 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 10:43:43.811242 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 10:43:43.811492 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":136,"threads_started":1}
I20260927 10:43:43.813988 2318 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36115
I20260927 10:43:43.814170 2318 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:43.814600 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 2187
I20260927 10:43:43.814718 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:43:43.814801 2318 heartbeater.cc:511] Master 127.0.98.62:36115 requested a full tablet report, sending...
I20260927 10:43:43.815096 392 external_mini_cluster.cc:1428] Setting key e9587d04b613f9ace226f12ff8db8ac5
I20260927 10:43:43.816469 2115 ts_manager.cc:194] Registered new tserver with Master: 16badd963c564d45ad782203936af1ae (127.0.98.1:44899)
I20260927 10:43:43.817601 2115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51461
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 10:43:43.826152 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:43.816556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53403 (local address 127.0.98.62:36115)
0927 10:43:43.816675 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:43.816679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:43.818399 (+ 1720us) server_negotiation.cc:407] Connection header received
0927 10:43:43.818580 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:43.818583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:43.818629 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:43.818718 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:43.819346 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:43.820065 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:43.821382 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:43.821600 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:43.821687 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:43.823106 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:43.823110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:43.823112 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:43.823133 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:43.824671 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:43.825087 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:43.825089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:43.825091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:43.825159 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:43.825402 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:43.825404 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:43.825405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:43.825566 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:43.825642 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:43.825949 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 10:43:43.826041 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:43:43.826152 2324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:43.816401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36115 (local address 127.0.98.1:53403)
0927 10:43:43.818281 (+ 1880us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:43.818301 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:43.818470 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:43.818754 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:43.818758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:43.819010 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:43.819205 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:43.819215 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:43.820201 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:43.820208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:43.821229 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:43.821242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:43.821724 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:43.821731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:43.821860 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:43.822450 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:43.822470 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:43.822960 (+ 490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:43.824777 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:43.824784 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:43.824789 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:43.825002 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:43.825233 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:43.825236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:43.825237 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:43.825306 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:43.825708 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:43.825710 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:43.825798 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 10:43:43.825881 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 10:43:43.825990 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1774,"mutex_wait_us":7,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 10:43:43.846624 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36115
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:43.957119 2327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:43.957388 2327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:43.957444 2327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:43.961473 2327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:43.961535 2327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:43.961576 2327 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 10:43:43.961591 2327 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:43:43.961622 2327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:43:43.966912 2327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:43.967976 2327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:43.969026 2327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:43.978833 2333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:43.978891 2335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:43.978868 2332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:43.978991 2327 server_base.cc:1034] running on GCE node
I20260927 10:43:43.979535 2327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:43.980238 2327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:43.981431 2327 hybrid_clock.cc:648] HybridClock initialized: now 1790505823981407 us; error 36 us; skew 500 ppm
Sep 27 10:43:43 dist-test-slave-dzhw krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:43.984337 2327 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 10:43:43.985173 2327 webserver.cc:460] Webserver started at http://127.0.98.2:35337/ using document root <none> and password file <none>
I20260927 10:43:43.985708 2327 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:43.985765 2327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:43.985930 2327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:43.988034 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "3440908a4a4f4d7190c56635dbd7c4ca"
format_stamp: "Formatted at 2026-09-27 10:43:43 on dist-test-slave-dzhw"
server_key: "4cd53103cf1b51788d70e6359acfe603"
server_key_iv: "ebd4be8f9d733fabcc914b519199a290"
server_key_version: "encryptionkey@0"
I20260927 10:43:43.988554 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3440908a4a4f4d7190c56635dbd7c4ca"
format_stamp: "Formatted at 2026-09-27 10:43:43 on dist-test-slave-dzhw"
server_key: "4cd53103cf1b51788d70e6359acfe603"
server_key_iv: "ebd4be8f9d733fabcc914b519199a290"
server_key_version: "encryptionkey@0"
I20260927 10:43:43.992344 2327 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 10:43:43.994910 2342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:43.996305 2327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:43:43.996444 2327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "3440908a4a4f4d7190c56635dbd7c4ca"
format_stamp: "Formatted at 2026-09-27 10:43:43 on dist-test-slave-dzhw"
server_key: "4cd53103cf1b51788d70e6359acfe603"
server_key_iv: "ebd4be8f9d733fabcc914b519199a290"
server_key_version: "encryptionkey@0"
I20260927 10:43:43.996570 2327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:44.020357 2327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:44.023692 2327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:44.023849 2327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:44.025235 2327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:44.025285 2327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:44.025324 2327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:44.025395 2327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:43:44 dist-test-slave-dzhw krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:44.036664 2327 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44795
I20260927 10:43:44.036710 2457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44795 every 8 connection(s)
I20260927 10:43:44.037539 2327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:43:44.039531 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.026278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50863 (local address 127.0.98.62:36115)
0927 10:43:44.026406 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:44.026409 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:44.027473 (+ 1064us) server_negotiation.cc:407] Connection header received
0927 10:43:44.028722 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:44.028726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:44.028793 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:44.028894 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:44.030550 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.031224 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.032540 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.032944 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.033077 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.035634 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:44.035638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:44.035641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:44.035668 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:44.037255 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:44.037863 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:44.037866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:44.037868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:44.037953 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:44.038313 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:44.038316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:44.038318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:44.038580 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:44.038688 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:44.039271 (+ 583us) server_negotiation.cc:299] Negotiation successful
0927 10:43:44.039404 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:43:44.040203 2351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.026866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36115 (local address 127.0.98.2:50863)
0927 10:43:44.027327 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:44.027369 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:44.028475 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:44.029153 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:44.029166 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:44.029633 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:44.030341 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.030362 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.031350 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.031354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.032417 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.032429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.033121 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.033126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.033342 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.033970 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:44.033988 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:44.035486 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:44.037377 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:44.037390 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:44.037404 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:44.037746 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:44.038047 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:44.038052 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:44.038054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:44.038193 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:44.038688 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:44.038693 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:44.038980 (+ 287us) client_negotiation.cc:769] Sending connection context
0927 10:43:44.039131 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 10:43:44.039376 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":122,"threads_started":1}
I20260927 10:43:44.041391 2458 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36115
I20260927 10:43:44.041575 2458 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:44.042080 2458 heartbeater.cc:511] Master 127.0.98.62:36115 requested a full tablet report, sending...
I20260927 10:43:44.043229 2114 ts_manager.cc:194] Registered new tserver with Master: 3440908a4a4f4d7190c56635dbd7c4ca (127.0.98.2:44795)
I20260927 10:43:44.043597 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 2327
I20260927 10:43:44.043685 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:43:44.043964 392 external_mini_cluster.cc:1428] Setting key 66ff1b29e5317b52a75acc1fb0e5cc29
I20260927 10:43:44.044082 2114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50863
I20260927 10:43:44.051915 2464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.043573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36115 (local address 127.0.98.2:44829)
0927 10:43:44.043822 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:44.043837 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:44.043927 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:44.044710 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:44.044713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:44.044955 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:44.045196 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.045206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.046063 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.046067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.047033 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.047042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.047515 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.047519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.047606 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.048151 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:44.048162 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:44.048510 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:44.050282 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:44.050287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:44.050289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:44.050566 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:44.050858 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:44.050862 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:44.050863 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:44.050948 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:44.051357 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:44.051361 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:44.051464 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 10:43:44.051572 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 10:43:44.051719 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":1,"thread_start_us":86,"threads_started":1}
I20260927 10:43:44.051916 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.044382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44829 (local address 127.0.98.62:36115)
0927 10:43:44.044495 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:44.044499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:44.044512 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:43:44.044552 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:44.044555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:44.044599 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:44.044693 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:44.045336 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.045907 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.047153 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.047399 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.047513 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.048645 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:44.048649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:44.048652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:44.048674 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:44.050169 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:44.050675 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:44.050679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:44.050680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:44.050761 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:44.051036 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:44.051041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:44.051043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:44.051258 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:44.051368 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:44.051625 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 10:43:44.051731 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 10:43:44.074961 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36115
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:44.190317 2467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:44.190665 2467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:44.190753 2467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:44.194941 2467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:44.195003 2467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:44.195075 2467 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 10:43:44.195104 2467 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:43:44.195166 2467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:43:44.200779 2467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:44.201936 2467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:44.203056 2467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:44.212911 2472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:44.213012 2467 server_base.cc:1034] running on GCE node
W20260927 10:43:44.212911 2473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:44.212915 2475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:44.213610 2467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:44.214476 2467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:44.215669 2467 hybrid_clock.cc:648] HybridClock initialized: now 1790505824215626 us; error 63 us; skew 500 ppm
Sep 27 10:43:44 dist-test-slave-dzhw krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:44.218616 2467 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 10:43:44.219645 2467 webserver.cc:460] Webserver started at http://127.0.98.3:37095/ using document root <none> and password file <none>
I20260927 10:43:44.220245 2467 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:44.220338 2467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:44.220613 2467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:44.222842 2467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "9fdba0a8ce2e480e8d8ecea0699bc830"
format_stamp: "Formatted at 2026-09-27 10:43:44 on dist-test-slave-dzhw"
server_key: "d36a7dc9f7422b73c1f4a8a056547d11"
server_key_iv: "98b03faec87cfa893dddd17cd4c8d59f"
server_key_version: "encryptionkey@0"
I20260927 10:43:44.223465 2467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9fdba0a8ce2e480e8d8ecea0699bc830"
format_stamp: "Formatted at 2026-09-27 10:43:44 on dist-test-slave-dzhw"
server_key: "d36a7dc9f7422b73c1f4a8a056547d11"
server_key_iv: "98b03faec87cfa893dddd17cd4c8d59f"
server_key_version: "encryptionkey@0"
I20260927 10:43:44.227510 2467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:43:44.230199 2482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:44.231616 2467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:43:44.231760 2467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "9fdba0a8ce2e480e8d8ecea0699bc830"
format_stamp: "Formatted at 2026-09-27 10:43:44 on dist-test-slave-dzhw"
server_key: "d36a7dc9f7422b73c1f4a8a056547d11"
server_key_iv: "98b03faec87cfa893dddd17cd4c8d59f"
server_key_version: "encryptionkey@0"
I20260927 10:43:44.231906 2467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:44.263485 2467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:44.267009 2467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:44.267226 2467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:44.268733 2467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:44.268810 2467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:44.268879 2467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:44.268932 2467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:43:44 dist-test-slave-dzhw krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:44.279927 2467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36219
I20260927 10:43:44.280831 2467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:43:44.281697 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 2467
I20260927 10:43:44.281769 2597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36219 every 8 connection(s)
I20260927 10:43:44.281816 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:43:44.282136 392 external_mini_cluster.cc:1428] Setting key f94057e3dd680159ebde828a7c7e573b
I20260927 10:43:44.285010 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.269797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40707 (local address 127.0.98.62:36115)
0927 10:43:44.269962 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:44.269967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:44.271035 (+ 1068us) server_negotiation.cc:407] Connection header received
0927 10:43:44.272425 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:44.272431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:44.272499 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:44.272608 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:44.274359 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.275057 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.277021 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.277235 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.277316 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.280479 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:44.280483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:44.280486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:44.280522 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:44.282989 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:44.283579 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:44.283583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:44.283585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:44.283671 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:44.283972 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:44.283975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:44.283977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:44.284192 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:44.284308 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:44.284784 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 10:43:44.284896 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 10:43:44.285830 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.270234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36115 (local address 127.0.98.3:40707)
0927 10:43:44.270824 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:44.270900 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:44.272123 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:44.272838 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:44.272850 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:44.273375 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:44.274195 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.274219 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.275205 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.275211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.276866 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.276876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.277397 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.277403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.277625 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.278461 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:44.278482 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:44.280280 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:44.283114 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:44.283118 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:44.283130 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:44.283496 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:44.283757 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:44.283759 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:44.283760 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:44.283885 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:44.284320 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:44.284325 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:44.284578 (+ 253us) client_negotiation.cc:769] Sending connection context
0927 10:43:44.284735 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 10:43:44.284989 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":156,"threads_started":1}
I20260927 10:43:44.286948 2598 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36115
I20260927 10:43:44.287083 2598 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:44.287672 2598 heartbeater.cc:511] Master 127.0.98.62:36115 requested a full tablet report, sending...
I20260927 10:43:44.288724 2115 ts_manager.cc:194] Registered new tserver with Master: 9fdba0a8ce2e480e8d8ecea0699bc830 (127.0.98.3:36219)
I20260927 10:43:44.289553 2115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40707
I20260927 10:43:44.296280 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:43:44.296480 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.289292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36115 (local address 127.0.98.3:60017)
0927 10:43:44.289551 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:44.289563 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:44.289669 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:44.289962 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:44.289964 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:44.290163 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:44.290287 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.290293 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.291064 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.291068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.291797 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.291805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.292183 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.292185 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.292247 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.292759 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:44.292779 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:44.293109 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:44.294722 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:44.294725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:44.294727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:44.294963 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:44.295431 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:44.295433 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:44.295435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:44.295514 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:44.295928 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:44.295931 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:44.296032 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 10:43:44.296116 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 10:43:44.296272 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":84,"threads_started":1}
I20260927 10:43:44.296487 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.289376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60017 (local address 127.0.98.62:36115)
0927 10:43:44.289497 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:44.289501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:44.289617 (+ 116us) server_negotiation.cc:407] Connection header received
0927 10:43:44.289763 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:44.289769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:44.289821 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:44.289926 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:44.290397 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.290935 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.291900 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.292094 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.292166 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.293251 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:44.293254 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:44.293256 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:44.293276 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:44.294601 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:44.295064 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:44.295069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:44.295070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:44.295174 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:44.295654 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:44.295657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:44.295658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:44.295840 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:44.296054 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:44.296195 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 10:43:44.296305 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 10:43:44 dist-test-slave-dzhw krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 10:43:44 dist-test-slave-dzhw krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:44.323050 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.314671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41974 (local address 127.0.98.62:36115)
0927 10:43:44.314849 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:44.314856 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:44.314874 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:43:44.314933 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:44.314935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:44.314977 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:44.315063 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:44.315855 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.316378 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.317253 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.317483 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.317571 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.320126 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:44.320129 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:44.320131 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:44.320157 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:44.321529 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:44.321942 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:44.321944 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:44.321946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:44.322022 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:44.322258 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:44.322260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:44.322261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:44.322513 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:44.322616 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:44.322818 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 10:43:44.322928 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:43:44.326148 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:43:44.329413 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:43:44.345577 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.339647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51652 (local address 127.0.98.2:44795)
0927 10:43:44.340531 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:44.340546 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:44.340565 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:43:44.340632 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:44.340637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:44.340798 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:44.340962 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:43:44.341170 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.342135 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.344182 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.344977 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.345100 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.345251 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:44.345347 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 10:43:44.345432 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":118,"threads_started":1}
I20260927 10:43:44.345593 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.338920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44795 (local address 127.0.0.1:51652)
0927 10:43:44.340066 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:44.340081 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:44.340157 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:44.340939 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:44.340942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:44.340956 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:43:44.341078 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.341086 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.342274 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.342278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.344046 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.344059 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.345110 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.345114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.345202 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.345227 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:43:44.345306 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 10:43:44.345381 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1082,"mutex_wait_us":156,"thread_start_us":80,"threads_started":1}
I20260927 10:43:44.345621 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.338813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36219 (local address 127.0.0.1:44840)
0927 10:43:44.339755 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:44.339804 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:44.339960 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:44.340734 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:44.340739 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:44.340757 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:43:44.340955 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.340962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.342324 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.342326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.344051 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.344059 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.345188 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.345192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.345306 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.345329 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:43:44.345402 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 10:43:44.345480 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":124,"thread_start_us":80,"threads_started":1}
I20260927 10:43:44.345839 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.339760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49598 (local address 127.0.98.1:44899)
0927 10:43:44.340165 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:44.340172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:44.340508 (+ 336us) server_negotiation.cc:407] Connection header received
0927 10:43:44.340596 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:44.340602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:44.340865 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:44.340989 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:43:44.341387 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.342563 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.343900 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.344648 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.344763 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.344923 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:44.345564 (+ 641us) server_negotiation.cc:299] Negotiation successful
0927 10:43:44.345680 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":57,"threads_started":1}
I20260927 10:43:44.346069 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.338927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44840 (local address 127.0.98.3:36219)
0927 10:43:44.339363 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:44.339373 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:44.340112 (+ 739us) server_negotiation.cc:407] Connection header received
0927 10:43:44.340197 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:44.340203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:44.340439 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:44.340586 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:43:44.341097 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.342236 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.344205 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.344854 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.344963 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.345127 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:44.345792 (+ 665us) server_negotiation.cc:299] Negotiation successful
0927 10:43:44.345897 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":136,"threads_started":1}
I20260927 10:43:44.345440 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.339554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44899 (local address 127.0.0.1:49598)
0927 10:43:44.340282 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:44.340298 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:44.340405 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:44.341117 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:44.341121 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:44.341140 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:43:44.341248 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.341254 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.342718 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.342722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.343764 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.343773 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.344797 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.344801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.344939 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.344988 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 10:43:44.345105 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 10:43:44.345245 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":22,"thread_start_us":82,"threads_started":1}
I20260927 10:43:44.349164 2252 tablet_service.cc:1467] Processing CreateTablet for tablet ff1b327e02e84e44a9e608da08f38b7e (DEFAULT_TABLE table=test-table [id=7cb5dcc48a664ec2ab0e3f9de02cf2e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:44.349378 2392 tablet_service.cc:1467] Processing CreateTablet for tablet ff1b327e02e84e44a9e608da08f38b7e (DEFAULT_TABLE table=test-table [id=7cb5dcc48a664ec2ab0e3f9de02cf2e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:44.350302 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff1b327e02e84e44a9e608da08f38b7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:44.350351 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff1b327e02e84e44a9e608da08f38b7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:44.351727 2532 tablet_service.cc:1467] Processing CreateTablet for tablet ff1b327e02e84e44a9e608da08f38b7e (DEFAULT_TABLE table=test-table [id=7cb5dcc48a664ec2ab0e3f9de02cf2e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:44.352718 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff1b327e02e84e44a9e608da08f38b7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:44.356439 2621 tablet_bootstrap.cc:492] T ff1b327e02e84e44a9e608da08f38b7e P 16badd963c564d45ad782203936af1ae: Bootstrap starting.
I20260927 10:43:44.358263 2622 tablet_bootstrap.cc:492] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca: Bootstrap starting.
I20260927 10:43:44.358791 2621 tablet_bootstrap.cc:654] T ff1b327e02e84e44a9e608da08f38b7e P 16badd963c564d45ad782203936af1ae: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:44.359766 2621 log.cc:824] T ff1b327e02e84e44a9e608da08f38b7e P 16badd963c564d45ad782203936af1ae: Log is configured to *not* fsync() on all Append() calls
I20260927 10:43:44.360065 2623 tablet_bootstrap.cc:492] T ff1b327e02e84e44a9e608da08f38b7e P 9fdba0a8ce2e480e8d8ecea0699bc830: Bootstrap starting.
I20260927 10:43:44.360498 2622 tablet_bootstrap.cc:654] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:44.361384 2622 log.cc:824] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca: Log is configured to *not* fsync() on all Append() calls
I20260927 10:43:44.362035 2623 tablet_bootstrap.cc:654] T ff1b327e02e84e44a9e608da08f38b7e P 9fdba0a8ce2e480e8d8ecea0699bc830: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:44.362309 2621 tablet_bootstrap.cc:492] T ff1b327e02e84e44a9e608da08f38b7e P 16badd963c564d45ad782203936af1ae: No bootstrap required, opened a new log
I20260927 10:43:44.362519 2621 ts_tablet_manager.cc:1397] T ff1b327e02e84e44a9e608da08f38b7e P 16badd963c564d45ad782203936af1ae: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:43:44.362808 2623 log.cc:824] T ff1b327e02e84e44a9e608da08f38b7e P 9fdba0a8ce2e480e8d8ecea0699bc830: Log is configured to *not* fsync() on all Append() calls
I20260927 10:43:44.363401 2622 tablet_bootstrap.cc:492] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca: No bootstrap required, opened a new log
I20260927 10:43:44.363605 2622 ts_tablet_manager.cc:1397] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:43:44.364521 2623 tablet_bootstrap.cc:492] T ff1b327e02e84e44a9e608da08f38b7e P 9fdba0a8ce2e480e8d8ecea0699bc830: No bootstrap required, opened a new log
I20260927 10:43:44.364699 2623 ts_tablet_manager.cc:1397] T ff1b327e02e84e44a9e608da08f38b7e P 9fdba0a8ce2e480e8d8ecea0699bc830: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:43:44.367213 2621 raft_consensus.cc:348] T ff1b327e02e84e44a9e608da08f38b7e P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } }
I20260927 10:43:44.367529 2621 raft_consensus.cc:374] T ff1b327e02e84e44a9e608da08f38b7e P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:44.367614 2621 raft_consensus.cc:729] T ff1b327e02e84e44a9e608da08f38b7e P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16badd963c564d45ad782203936af1ae, State: Initialized, Role: FOLLOWER
I20260927 10:43:44.367434 2622 raft_consensus.cc:348] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } }
I20260927 10:43:44.367687 2622 raft_consensus.cc:374] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:44.367769 2622 raft_consensus.cc:729] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3440908a4a4f4d7190c56635dbd7c4ca, State: Initialized, Role: FOLLOWER
I20260927 10:43:44.367661 2623 raft_consensus.cc:348] T ff1b327e02e84e44a9e608da08f38b7e P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } }
I20260927 10:43:44.367942 2623 raft_consensus.cc:374] T ff1b327e02e84e44a9e608da08f38b7e P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:44.368000 2623 raft_consensus.cc:729] T ff1b327e02e84e44a9e608da08f38b7e P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fdba0a8ce2e480e8d8ecea0699bc830, State: Initialized, Role: FOLLOWER
I20260927 10:43:44.368188 2621 consensus_queue.cc:260] T ff1b327e02e84e44a9e608da08f38b7e P 16badd963c564d45ad782203936af1ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } }
I20260927 10:43:44.368188 2622 consensus_queue.cc:260] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } }
I20260927 10:43:44.368445 2623 consensus_queue.cc:260] T ff1b327e02e84e44a9e608da08f38b7e P 9fdba0a8ce2e480e8d8ecea0699bc830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } }
I20260927 10:43:44.368958 2318 heartbeater.cc:503] Master 127.0.98.62:36115 was elected leader, sending a full tablet report...
I20260927 10:43:44.369215 2621 ts_tablet_manager.cc:1428] T ff1b327e02e84e44a9e608da08f38b7e P 16badd963c564d45ad782203936af1ae: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 10:43:44.369215 2622 ts_tablet_manager.cc:1428] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:43:44.369550 2458 heartbeater.cc:503] Master 127.0.98.62:36115 was elected leader, sending a full tablet report...
I20260927 10:43:44.370560 2598 heartbeater.cc:503] Master 127.0.98.62:36115 was elected leader, sending a full tablet report...
I20260927 10:43:44.370625 2623 ts_tablet_manager.cc:1428] T ff1b327e02e84e44a9e608da08f38b7e P 9fdba0a8ce2e480e8d8ecea0699bc830: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:43:44.415699 2628 raft_consensus.cc:482] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:43:44.415923 2628 raft_consensus.cc:504] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } }
I20260927 10:43:44.417096 2628 leader_election.cc:288] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fdba0a8ce2e480e8d8ecea0699bc830 (127.0.98.3:36219), 16badd963c564d45ad782203936af1ae (127.0.98.1:44899)
I20260927 10:43:44.421602 2351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.417486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36219 (local address 127.0.98.2:38531)
0927 10:43:44.417651 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:44.417669 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:44.417791 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:44.418137 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:44.418140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:44.418158 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:43:44.418293 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.418299 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.419171 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.419177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.420180 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.420190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.421103 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.421107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.421216 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.421240 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 10:43:44.421324 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 10:43:44.421435 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 10:43:44.421948 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.417955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41383 (local address 127.0.98.1:44899)
0927 10:43:44.418101 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:44.418105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:44.418267 (+ 162us) server_negotiation.cc:407] Connection header received
0927 10:43:44.418445 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:44.418448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:44.418499 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:44.418584 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:43:44.418854 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.419344 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.420743 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.421303 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.421417 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.421474 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:44.421697 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 10:43:44.421793 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 10:43:44.422132 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.417560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38531 (local address 127.0.98.3:36219)
0927 10:43:44.417710 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:44.417716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:44.417734 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:43:44.417933 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:44.417941 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:44.418015 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:44.418109 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:43:44.418439 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.419041 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.420331 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.420843 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.420958 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.421017 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:44.421918 (+ 901us) server_negotiation.cc:299] Negotiation successful
0927 10:43:44.422006 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 10:43:44.422330 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.417788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44899 (local address 127.0.98.2:41383)
0927 10:43:44.418177 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:44.418189 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:44.418302 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:44.418596 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:44.418599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:44.418615 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:43:44.418746 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.418751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.419462 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.419468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.420611 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.420620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.421431 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.421435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.421534 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.421555 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:43:44.422090 (+ 535us) client_negotiation.cc:240] Negotiation successful
0927 10:43:44.422172 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":164,"threads_started":1}
I20260927 10:43:44.422735 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff1b327e02e84e44a9e608da08f38b7e" candidate_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" is_pre_election: true
I20260927 10:43:44.423063 2552 raft_consensus.cc:2454] T ff1b327e02e84e44a9e608da08f38b7e P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3440908a4a4f4d7190c56635dbd7c4ca in term 0.
I20260927 10:43:44.423084 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff1b327e02e84e44a9e608da08f38b7e" candidate_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16badd963c564d45ad782203936af1ae" is_pre_election: true
I20260927 10:43:44.423378 2272 raft_consensus.cc:2454] T ff1b327e02e84e44a9e608da08f38b7e P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3440908a4a4f4d7190c56635dbd7c4ca in term 0.
I20260927 10:43:44.423601 2343 leader_election.cc:302] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3440908a4a4f4d7190c56635dbd7c4ca, 9fdba0a8ce2e480e8d8ecea0699bc830; no voters:
I20260927 10:43:44.423929 2628 raft_consensus.cc:2781] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:43:44.424026 2628 raft_consensus.cc:482] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:43:44.424088 2628 raft_consensus.cc:3037] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:44.425135 2628 raft_consensus.cc:504] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } }
I20260927 10:43:44.425482 2628 leader_election.cc:288] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca [CANDIDATE]: Term 1 election: Requested vote from peers 9fdba0a8ce2e480e8d8ecea0699bc830 (127.0.98.3:36219), 16badd963c564d45ad782203936af1ae (127.0.98.1:44899)
I20260927 10:43:44.425872 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff1b327e02e84e44a9e608da08f38b7e" candidate_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830"
I20260927 10:43:44.425936 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff1b327e02e84e44a9e608da08f38b7e" candidate_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16badd963c564d45ad782203936af1ae"
I20260927 10:43:44.425985 2552 raft_consensus.cc:3037] T ff1b327e02e84e44a9e608da08f38b7e P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:44.426085 2272 raft_consensus.cc:3037] T ff1b327e02e84e44a9e608da08f38b7e P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:44.427126 2272 raft_consensus.cc:2454] T ff1b327e02e84e44a9e608da08f38b7e P 16badd963c564d45ad782203936af1ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3440908a4a4f4d7190c56635dbd7c4ca in term 1.
I20260927 10:43:44.427332 2552 raft_consensus.cc:2454] T ff1b327e02e84e44a9e608da08f38b7e P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3440908a4a4f4d7190c56635dbd7c4ca in term 1.
I20260927 10:43:44.427488 2343 leader_election.cc:302] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16badd963c564d45ad782203936af1ae, 3440908a4a4f4d7190c56635dbd7c4ca; no voters:
I20260927 10:43:44.427696 2628 raft_consensus.cc:2781] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:43:44.427955 2628 raft_consensus.cc:686] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca [term 1 LEADER]: Becoming Leader. State: Replica: 3440908a4a4f4d7190c56635dbd7c4ca, State: Running, Role: LEADER
I20260927 10:43:44.428324 2628 consensus_queue.cc:237] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } }
I20260927 10:43:44.431665 2114 catalog_manager.cc:5495] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 3440908a4a4f4d7190c56635dbd7c4ca (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } health_report { overall_health: UNKNOWN } } }
Sep 27 10:43:44 dist-test-slave-dzhw krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:44.481873 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.472339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36115 (local address 127.0.0.1:41984)
0927 10:43:44.472888 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:44.472901 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:44.473042 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:44.473283 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:44.473285 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:44.473737 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:44.473861 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.473867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.474713 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.474718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.475278 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.475286 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.475812 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.475815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.475938 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.476576 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:44.476588 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:44.478187 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:44.480270 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:44.480279 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:44.480286 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:44.480547 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:44.480834 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:44.480839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:44.480841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:44.480933 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:44.481322 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:44.481328 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:44.481468 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 10:43:44.481572 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 10:43:44.481711 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":20,"thread_start_us":168,"threads_started":1}
I20260927 10:43:44.481900 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.472485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41984 (local address 127.0.98.62:36115)
0927 10:43:44.472820 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:44.472826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:44.472978 (+ 152us) server_negotiation.cc:407] Connection header received
0927 10:43:44.473134 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:44.473137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:44.473174 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:44.473268 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:44.473979 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.474608 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.475411 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.475696 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.475806 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.478346 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:44.478349 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:44.478352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:44.478374 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:44.480153 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:44.480637 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:44.480640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:44.480641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:44.480724 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:44.481021 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:44.481024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:44.481025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:44.481205 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:44.481280 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:44.481628 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 10:43:44.481790 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":242}
I20260927 10:43:44.486006 2114 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41984:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 10:43:44.486784 2114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:43:44.491811 2252 tablet_service.cc:1467] Processing CreateTablet for tablet 7e083515f97d4883926d68ad8109c78a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ffa30ab9f904f8ba16240ac94631a5f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:43:44.491809 2392 tablet_service.cc:1467] Processing CreateTablet for tablet 7e083515f97d4883926d68ad8109c78a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ffa30ab9f904f8ba16240ac94631a5f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:43:44.492628 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e083515f97d4883926d68ad8109c78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:44.492628 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e083515f97d4883926d68ad8109c78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:44.492502 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 7e083515f97d4883926d68ad8109c78a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ffa30ab9f904f8ba16240ac94631a5f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:43:44.493134 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e083515f97d4883926d68ad8109c78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:44.495698 2621 tablet_bootstrap.cc:492] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae: Bootstrap starting.
I20260927 10:43:44.495960 2622 tablet_bootstrap.cc:492] T 7e083515f97d4883926d68ad8109c78a P 3440908a4a4f4d7190c56635dbd7c4ca: Bootstrap starting.
I20260927 10:43:44.496186 2623 tablet_bootstrap.cc:492] T 7e083515f97d4883926d68ad8109c78a P 9fdba0a8ce2e480e8d8ecea0699bc830: Bootstrap starting.
I20260927 10:43:44.496789 2621 tablet_bootstrap.cc:654] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:44.496905 2622 tablet_bootstrap.cc:654] T 7e083515f97d4883926d68ad8109c78a P 3440908a4a4f4d7190c56635dbd7c4ca: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:44.497205 2623 tablet_bootstrap.cc:654] T 7e083515f97d4883926d68ad8109c78a P 9fdba0a8ce2e480e8d8ecea0699bc830: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:44.498281 2622 tablet_bootstrap.cc:492] T 7e083515f97d4883926d68ad8109c78a P 3440908a4a4f4d7190c56635dbd7c4ca: No bootstrap required, opened a new log
I20260927 10:43:44.498380 2622 ts_tablet_manager.cc:1397] T 7e083515f97d4883926d68ad8109c78a P 3440908a4a4f4d7190c56635dbd7c4ca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 10:43:44.498782 2621 tablet_bootstrap.cc:492] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae: No bootstrap required, opened a new log
I20260927 10:43:44.498874 2621 ts_tablet_manager.cc:1397] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:43:44.498955 2622 raft_consensus.cc:348] T 7e083515f97d4883926d68ad8109c78a P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:44.499096 2622 raft_consensus.cc:374] T 7e083515f97d4883926d68ad8109c78a P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:44.499109 2623 tablet_bootstrap.cc:492] T 7e083515f97d4883926d68ad8109c78a P 9fdba0a8ce2e480e8d8ecea0699bc830: No bootstrap required, opened a new log
I20260927 10:43:44.499131 2622 raft_consensus.cc:729] T 7e083515f97d4883926d68ad8109c78a P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3440908a4a4f4d7190c56635dbd7c4ca, State: Initialized, Role: FOLLOWER
I20260927 10:43:44.499197 2623 ts_tablet_manager.cc:1397] T 7e083515f97d4883926d68ad8109c78a P 9fdba0a8ce2e480e8d8ecea0699bc830: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:43:44.499289 2622 consensus_queue.cc:260] T 7e083515f97d4883926d68ad8109c78a P 3440908a4a4f4d7190c56635dbd7c4ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:44.499361 2621 raft_consensus.cc:348] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:44.499477 2621 raft_consensus.cc:374] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:44.499528 2621 raft_consensus.cc:729] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16badd963c564d45ad782203936af1ae, State: Initialized, Role: FOLLOWER
I20260927 10:43:44.499583 2628 tablet_replica.cc:439] T 7e083515f97d4883926d68ad8109c78a P 3440908a4a4f4d7190c56635dbd7c4ca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } } }
I20260927 10:43:44.499650 2621 consensus_queue.cc:260] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:44.499722 2628 tablet_replica.cc:442] T 7e083515f97d4883926d68ad8109c78a P 3440908a4a4f4d7190c56635dbd7c4ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:43:44.499717 2623 raft_consensus.cc:348] T 7e083515f97d4883926d68ad8109c78a P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:44.499866 2623 raft_consensus.cc:374] T 7e083515f97d4883926d68ad8109c78a P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:44.499909 2623 raft_consensus.cc:729] T 7e083515f97d4883926d68ad8109c78a P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fdba0a8ce2e480e8d8ecea0699bc830, State: Initialized, Role: FOLLOWER
I20260927 10:43:44.499931 2627 tablet_replica.cc:439] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } } }
I20260927 10:43:44.500067 2627 tablet_replica.cc:442] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:43:44.500052 2623 consensus_queue.cc:260] T 7e083515f97d4883926d68ad8109c78a P 9fdba0a8ce2e480e8d8ecea0699bc830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:44.500339 2629 tablet_replica.cc:439] T 7e083515f97d4883926d68ad8109c78a P 9fdba0a8ce2e480e8d8ecea0699bc830: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } } }
I20260927 10:43:44.500478 2629 tablet_replica.cc:442] T 7e083515f97d4883926d68ad8109c78a P 9fdba0a8ce2e480e8d8ecea0699bc830: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:43:44.500720 2623 ts_tablet_manager.cc:1428] T 7e083515f97d4883926d68ad8109c78a P 9fdba0a8ce2e480e8d8ecea0699bc830: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:43:44.500795 2622 ts_tablet_manager.cc:1428] T 7e083515f97d4883926d68ad8109c78a P 3440908a4a4f4d7190c56635dbd7c4ca: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260927 10:43:44.501299 2621 ts_tablet_manager.cc:1428] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260927 10:43:44.532136 2599 tablet.cc:2369] T 7e083515f97d4883926d68ad8109c78a P 9fdba0a8ce2e480e8d8ecea0699bc830: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:43:44.539144 2459 tablet.cc:2369] T 7e083515f97d4883926d68ad8109c78a P 3440908a4a4f4d7190c56635dbd7c4ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:43:44.561074 2319 tablet.cc:2369] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:43:44.789427 2627 raft_consensus.cc:482] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:43:44.789567 2627 raft_consensus.cc:504] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:44.790803 2627 leader_election.cc:288] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3440908a4a4f4d7190c56635dbd7c4ca (127.0.98.2:44795), 9fdba0a8ce2e480e8d8ecea0699bc830 (127.0.98.3:36219)
I20260927 10:43:44.795265 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.791136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40959 (local address 127.0.98.3:36219)
0927 10:43:44.791270 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:44.791275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:44.791553 (+ 278us) server_negotiation.cc:407] Connection header received
0927 10:43:44.791678 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:44.791684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:44.791740 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:44.791796 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:43:44.792130 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.792865 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.794105 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.794684 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.794759 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.794811 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:44.795063 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 10:43:44.795120 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:43:44.795557 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.791052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36219 (local address 127.0.98.1:40959)
0927 10:43:44.791446 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:44.791458 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:44.791579 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:44.791844 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:44.791847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:44.791862 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:43:44.792012 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.792018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.792971 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.792975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.793979 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.793989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.794807 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.794811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.794930 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.794956 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 10:43:44.795044 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 10:43:44.795120 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":1,"spinlock_wait_cycles":16640,"thread_start_us":147,"threads_started":1}
I20260927 10:43:44.795697 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.791217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46707 (local address 127.0.98.2:44795)
0927 10:43:44.791346 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:44.791351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:44.791587 (+ 236us) server_negotiation.cc:407] Connection header received
0927 10:43:44.791715 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:44.791717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:44.791759 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:44.791852 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:43:44.792664 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.793296 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.794533 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.795139 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.795245 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.795301 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:44.795506 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 10:43:44.795581 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:43:44.795722 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.791052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44795 (local address 127.0.98.1:46707)
0927 10:43:44.791460 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:44.791472 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:44.791607 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:44.791882 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:44.791885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:44.791923 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:43:44.792534 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.792545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.793426 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.793429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.794373 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.794382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.795256 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.795260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.795383 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.795407 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 10:43:44.795487 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 10:43:44.795550 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":26,"spinlock_wait_cycles":768,"thread_start_us":124,"threads_started":1}
I20260927 10:43:44.796073 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e083515f97d4883926d68ad8109c78a" candidate_uuid: "16badd963c564d45ad782203936af1ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" is_pre_election: true
I20260927 10:43:44.796235 2552 raft_consensus.cc:2454] T 7e083515f97d4883926d68ad8109c78a P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16badd963c564d45ad782203936af1ae in term 0.
I20260927 10:43:44.796320 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e083515f97d4883926d68ad8109c78a" candidate_uuid: "16badd963c564d45ad782203936af1ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" is_pre_election: true
I20260927 10:43:44.796586 2412 raft_consensus.cc:2454] T 7e083515f97d4883926d68ad8109c78a P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16badd963c564d45ad782203936af1ae in term 0.
I20260927 10:43:44.796623 2203 leader_election.cc:302] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16badd963c564d45ad782203936af1ae, 9fdba0a8ce2e480e8d8ecea0699bc830; no voters:
I20260927 10:43:44.796892 2627 raft_consensus.cc:2781] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:43:44.796954 2627 raft_consensus.cc:482] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:43:44.797026 2627 raft_consensus.cc:3037] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:44.798023 2627 raft_consensus.cc:504] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:44.798489 2627 leader_election.cc:288] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae [CANDIDATE]: Term 1 election: Requested vote from peers 3440908a4a4f4d7190c56635dbd7c4ca (127.0.98.2:44795), 9fdba0a8ce2e480e8d8ecea0699bc830 (127.0.98.3:36219)
I20260927 10:43:44.798810 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e083515f97d4883926d68ad8109c78a" candidate_uuid: "16badd963c564d45ad782203936af1ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3440908a4a4f4d7190c56635dbd7c4ca"
I20260927 10:43:44.798813 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e083515f97d4883926d68ad8109c78a" candidate_uuid: "16badd963c564d45ad782203936af1ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830"
I20260927 10:43:44.798933 2552 raft_consensus.cc:3037] T 7e083515f97d4883926d68ad8109c78a P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:44.798933 2412 raft_consensus.cc:3037] T 7e083515f97d4883926d68ad8109c78a P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:44.800033 2412 raft_consensus.cc:2454] T 7e083515f97d4883926d68ad8109c78a P 3440908a4a4f4d7190c56635dbd7c4ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16badd963c564d45ad782203936af1ae in term 1.
I20260927 10:43:44.800033 2552 raft_consensus.cc:2454] T 7e083515f97d4883926d68ad8109c78a P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16badd963c564d45ad782203936af1ae in term 1.
I20260927 10:43:44.800321 2203 leader_election.cc:302] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16badd963c564d45ad782203936af1ae, 9fdba0a8ce2e480e8d8ecea0699bc830; no voters:
I20260927 10:43:44.800551 2627 raft_consensus.cc:2781] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:43:44.800776 2627 raft_consensus.cc:686] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae [term 1 LEADER]: Becoming Leader. State: Replica: 16badd963c564d45ad782203936af1ae, State: Running, Role: LEADER
I20260927 10:43:44.801137 2627 consensus_queue.cc:237] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:44.803510 2644 tablet_replica.cc:439] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae: TxnStatusTablet state changed. Reason: New leader 16badd963c564d45ad782203936af1ae. Latest consensus state: current_term: 1 leader_uuid: "16badd963c564d45ad782203936af1ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } } }
I20260927 10:43:44.803622 2644 tablet_replica.cc:442] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:43:44.803890 2646 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 10:43:44.804093 2646 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 10:43:44.804744 2113 catalog_manager.cc:5495] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 16badd963c564d45ad782203936af1ae (127.0.98.1). New cstate: current_term: 1 leader_uuid: "16badd963c564d45ad782203936af1ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } health_report { overall_health: UNKNOWN } } }
I20260927 10:43:44.869705 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.865294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44899 (local address 127.0.0.1:49610)
0927 10:43:44.865508 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:44.865527 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:44.865639 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:44.865987 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:44.865990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:44.865997 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:43:44.866091 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.866096 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.867163 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.867166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.867792 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.867802 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.868410 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.868413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.868488 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.868535 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:43:44.869409 (+ 874us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:43:44.869429 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:43:44.869509 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 10:43:44.869587 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 10:43:44.869758 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.865491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49610 (local address 127.0.98.1:44899)
0927 10:43:44.865640 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:44.865645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:44.865661 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:43:44.865766 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:44.865769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:44.865840 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:44.865946 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:43:44.866202 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.867040 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.867914 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.868292 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.868403 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.868681 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:43:44.869259 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:43:44.869370 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:44.869543 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 10:43:44.869605 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 10:43:44.875347 2412 raft_consensus.cc:1261] T 7e083515f97d4883926d68ad8109c78a P 3440908a4a4f4d7190c56635dbd7c4ca [term 1 FOLLOWER]: Refusing update from remote peer 16badd963c564d45ad782203936af1ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:43:44.875422 2552 raft_consensus.cc:1261] T 7e083515f97d4883926d68ad8109c78a P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 1 FOLLOWER]: Refusing update from remote peer 16badd963c564d45ad782203936af1ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:43:44.876052 2644 consensus_queue.cc:1035] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:43:44.876269 2627 consensus_queue.cc:1035] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:43:44.879535 2628 tablet_replica.cc:439] T 7e083515f97d4883926d68ad8109c78a P 3440908a4a4f4d7190c56635dbd7c4ca: TxnStatusTablet state changed. Reason: New leader 16badd963c564d45ad782203936af1ae. Latest consensus state: current_term: 1 leader_uuid: "16badd963c564d45ad782203936af1ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } } }
I20260927 10:43:44.879667 2628 tablet_replica.cc:442] T 7e083515f97d4883926d68ad8109c78a P 3440908a4a4f4d7190c56635dbd7c4ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:43:44.880316 2629 tablet_replica.cc:439] T 7e083515f97d4883926d68ad8109c78a P 9fdba0a8ce2e480e8d8ecea0699bc830: TxnStatusTablet state changed. Reason: New leader 16badd963c564d45ad782203936af1ae. Latest consensus state: current_term: 1 leader_uuid: "16badd963c564d45ad782203936af1ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } } }
I20260927 10:43:44.880447 2629 tablet_replica.cc:442] T 7e083515f97d4883926d68ad8109c78a P 9fdba0a8ce2e480e8d8ecea0699bc830: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:43:44.883476 2644 tablet_replica.cc:439] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16badd963c564d45ad782203936af1ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } } }
I20260927 10:43:44.883601 2644 tablet_replica.cc:442] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:43:44.883697 2649 mvcc.cc:205] Tried to move back new op lower bound from 7333911858682064896 to 7333911858388226048. Current Snapshot: MvccSnapshot[applied={T|T < 7333911858682064896}]
I20260927 10:43:44.885347 2628 tablet_replica.cc:439] T 7e083515f97d4883926d68ad8109c78a P 3440908a4a4f4d7190c56635dbd7c4ca: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16badd963c564d45ad782203936af1ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } } }
I20260927 10:43:44.885471 2628 tablet_replica.cc:442] T 7e083515f97d4883926d68ad8109c78a P 3440908a4a4f4d7190c56635dbd7c4ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:43:44.889001 2627 tablet_replica.cc:439] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16badd963c564d45ad782203936af1ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } } }
I20260927 10:43:44.889127 2627 tablet_replica.cc:442] T 7e083515f97d4883926d68ad8109c78a P 16badd963c564d45ad782203936af1ae: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:43:44.893081 2629 tablet_replica.cc:439] T 7e083515f97d4883926d68ad8109c78a P 9fdba0a8ce2e480e8d8ecea0699bc830: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16badd963c564d45ad782203936af1ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } } }
I20260927 10:43:44.893214 2629 tablet_replica.cc:442] T 7e083515f97d4883926d68ad8109c78a P 9fdba0a8ce2e480e8d8ecea0699bc830: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:43:44.893513 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.889355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51660 (local address 127.0.98.2:44795)
0927 10:43:44.889472 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:44.889476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:44.889722 (+ 246us) server_negotiation.cc:407] Connection header received
0927 10:43:44.889902 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:44.889905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:44.889953 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:44.890059 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:43:44.890305 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.890890 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.891895 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.892125 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.892237 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.892533 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:43:44.893051 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:43:44.893162 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:44.893315 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 10:43:44.893386 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:43:44.904902 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.901328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51081 (local address 127.0.98.1:44899)
0927 10:43:44.901463 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:44.901467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:44.901638 (+ 171us) server_negotiation.cc:407] Connection header received
0927 10:43:44.901747 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:44.901750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:44.901805 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:44.901890 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:43:44.902107 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.902879 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.903719 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.904053 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.904162 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.904454 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:43:44.904536 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:43:44.904612 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:44.904697 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 10:43:44.904751 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 10:43:44.904954 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.901248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44899 (local address 127.0.98.2:51081)
0927 10:43:44.901542 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:44.901564 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:44.901655 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:44.901902 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:44.901904 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:44.901910 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:43:44.902012 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.902016 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.902988 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.902990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.903602 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.903610 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.904187 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.904190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.904256 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.904295 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:43:44.904627 (+ 332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:43:44.904632 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 10:43:44.904682 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 10:43:44.904798 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":126,"threads_started":1}
I20260927 10:43:44.910537 2628 consensus_queue.cc:1035] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:43:44.912853 2552 raft_consensus.cc:1261] T ff1b327e02e84e44a9e608da08f38b7e P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 1 FOLLOWER]: Refusing update from remote peer 3440908a4a4f4d7190c56635dbd7c4ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:43:44.913276 2628 consensus_queue.cc:1035] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:43:44.938678 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.934273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56139 (local address 127.0.98.2:44795)
0927 10:43:44.934406 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:44.934431 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:44.934643 (+ 212us) server_negotiation.cc:407] Connection header received
0927 10:43:44.934735 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:44.934738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:44.934819 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:44.934903 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:43:44.935272 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.935934 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.937389 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.937620 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.937711 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.938085 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:43:44.938196 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:43:44.938280 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:44.938455 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 10:43:44.938520 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:43:44.938700 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:44.934161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44795 (local address 127.0.98.1:56139)
0927 10:43:44.934548 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:44.934563 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:44.934655 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:44.935022 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:44.935025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:44.935034 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:43:44.935163 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.935171 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.936162 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.936166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.937233 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:44.937245 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:44.937774 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:44.937778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:44.937877 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:44.937941 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:43:44.938308 (+ 367us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:43:44.938316 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 10:43:44.938435 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 10:43:44.938523 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":91,"threads_started":1}
I20260927 10:43:44.966594 2115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:43:44.966802 2115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:43:44.969547 2115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2: Sending DeleteTablet for 3 replicas of tablet ff1b327e02e84e44a9e608da08f38b7e
I20260927 10:43:44.970351 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet ff1b327e02e84e44a9e608da08f38b7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:43:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51652
I20260927 10:43:44.970566 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet ff1b327e02e84e44a9e608da08f38b7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:43:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49598
I20260927 10:43:44.971000 2670 tablet_replica.cc:330] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca: stopping tablet replica
I20260927 10:43:44.971329 2670 raft_consensus.cc:2229] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:43:44.971285 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet ff1b327e02e84e44a9e608da08f38b7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:43:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44840
I20260927 10:43:44.971752 2670 raft_consensus.cc:2258] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:43:44.972471 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:43:44.972905 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:43:44.973434 2670 ts_tablet_manager.cc:1905] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:43:44.977046 2670 ts_tablet_manager.cc:1918] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:43:44.977218 2670 log.cc:1196] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-1/wal/wals/ff1b327e02e84e44a9e608da08f38b7e
I20260927 10:43:44.977629 2670 ts_tablet_manager.cc:1939] T ff1b327e02e84e44a9e608da08f38b7e P 3440908a4a4f4d7190c56635dbd7c4ca: Deleting consensus metadata
I20260927 10:43:44.978472 2392 tablet_service.cc:1467] Processing CreateTablet for tablet f2cb5bb17b1940f3acbd647e74b09d21 (DEFAULT_TABLE table=test-table [id=77e0d955747b43f8809c62dd59aec783]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:44.978561 2672 tablet_replica.cc:330] T ff1b327e02e84e44a9e608da08f38b7e P 9fdba0a8ce2e480e8d8ecea0699bc830: stopping tablet replica
I20260927 10:43:44.978724 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2cb5bb17b1940f3acbd647e74b09d21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:44.978894 2672 raft_consensus.cc:2229] T ff1b327e02e84e44a9e608da08f38b7e P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:43:44.979117 2672 raft_consensus.cc:2258] T ff1b327e02e84e44a9e608da08f38b7e P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:43:44.979135 2101 catalog_manager.cc:4841] TS 3440908a4a4f4d7190c56635dbd7c4ca (127.0.98.2:44795): tablet ff1b327e02e84e44a9e608da08f38b7e (table test-table [id=7cb5dcc48a664ec2ab0e3f9de02cf2e9]) successfully deleted
I20260927 10:43:44.979504 2671 tablet_replica.cc:330] T ff1b327e02e84e44a9e608da08f38b7e P 16badd963c564d45ad782203936af1ae: stopping tablet replica
I20260927 10:43:44.979786 2671 raft_consensus.cc:2229] T ff1b327e02e84e44a9e608da08f38b7e P 16badd963c564d45ad782203936af1ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:43:44.979964 2671 raft_consensus.cc:2258] T ff1b327e02e84e44a9e608da08f38b7e P 16badd963c564d45ad782203936af1ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:43:44.980866 2622 tablet_bootstrap.cc:492] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca: Bootstrap starting.
I20260927 10:43:44.981285 2671 ts_tablet_manager.cc:1905] T ff1b327e02e84e44a9e608da08f38b7e P 16badd963c564d45ad782203936af1ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:43:44.981563 2672 ts_tablet_manager.cc:1905] T ff1b327e02e84e44a9e608da08f38b7e P 9fdba0a8ce2e480e8d8ecea0699bc830: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:43:44.981758 2622 tablet_bootstrap.cc:654] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:44.982741 2532 tablet_service.cc:1467] Processing CreateTablet for tablet f2cb5bb17b1940f3acbd647e74b09d21 (DEFAULT_TABLE table=test-table [id=77e0d955747b43f8809c62dd59aec783]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:44.982992 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2cb5bb17b1940f3acbd647e74b09d21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:44.984542 2671 ts_tablet_manager.cc:1918] T ff1b327e02e84e44a9e608da08f38b7e P 16badd963c564d45ad782203936af1ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:43:44.984542 2672 ts_tablet_manager.cc:1918] T ff1b327e02e84e44a9e608da08f38b7e P 9fdba0a8ce2e480e8d8ecea0699bc830: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:43:44.984635 2672 log.cc:1196] T ff1b327e02e84e44a9e608da08f38b7e P 9fdba0a8ce2e480e8d8ecea0699bc830: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/ff1b327e02e84e44a9e608da08f38b7e
I20260927 10:43:44.984732 2671 log.cc:1196] T ff1b327e02e84e44a9e608da08f38b7e P 16badd963c564d45ad782203936af1ae: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-0/wal/wals/ff1b327e02e84e44a9e608da08f38b7e
I20260927 10:43:44.985109 2672 ts_tablet_manager.cc:1939] T ff1b327e02e84e44a9e608da08f38b7e P 9fdba0a8ce2e480e8d8ecea0699bc830: Deleting consensus metadata
I20260927 10:43:44.985270 2671 ts_tablet_manager.cc:1939] T ff1b327e02e84e44a9e608da08f38b7e P 16badd963c564d45ad782203936af1ae: Deleting consensus metadata
I20260927 10:43:44.985332 2623 tablet_bootstrap.cc:492] T f2cb5bb17b1940f3acbd647e74b09d21 P 9fdba0a8ce2e480e8d8ecea0699bc830: Bootstrap starting.
I20260927 10:43:44.986215 2623 tablet_bootstrap.cc:654] T f2cb5bb17b1940f3acbd647e74b09d21 P 9fdba0a8ce2e480e8d8ecea0699bc830: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:44.986435 2099 catalog_manager.cc:4841] TS 16badd963c564d45ad782203936af1ae (127.0.98.1:44899): tablet ff1b327e02e84e44a9e608da08f38b7e (table test-table [id=7cb5dcc48a664ec2ab0e3f9de02cf2e9]) successfully deleted
I20260927 10:43:44.986707 2099 catalog_manager.cc:4841] TS 9fdba0a8ce2e480e8d8ecea0699bc830 (127.0.98.3:36219): tablet ff1b327e02e84e44a9e608da08f38b7e (table test-table [id=7cb5dcc48a664ec2ab0e3f9de02cf2e9]) successfully deleted
I20260927 10:43:44.986792 2252 tablet_service.cc:1467] Processing CreateTablet for tablet f2cb5bb17b1940f3acbd647e74b09d21 (DEFAULT_TABLE table=test-table [id=77e0d955747b43f8809c62dd59aec783]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:44.987071 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2cb5bb17b1940f3acbd647e74b09d21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:44.987406 2623 tablet_bootstrap.cc:492] T f2cb5bb17b1940f3acbd647e74b09d21 P 9fdba0a8ce2e480e8d8ecea0699bc830: No bootstrap required, opened a new log
I20260927 10:43:44.987489 2623 ts_tablet_manager.cc:1397] T f2cb5bb17b1940f3acbd647e74b09d21 P 9fdba0a8ce2e480e8d8ecea0699bc830: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 10:43:44.987931 2623 raft_consensus.cc:348] T f2cb5bb17b1940f3acbd647e74b09d21 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:44.988045 2622 tablet_bootstrap.cc:492] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca: No bootstrap required, opened a new log
I20260927 10:43:44.988044 2623 raft_consensus.cc:374] T f2cb5bb17b1940f3acbd647e74b09d21 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:44.988093 2623 raft_consensus.cc:729] T f2cb5bb17b1940f3acbd647e74b09d21 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fdba0a8ce2e480e8d8ecea0699bc830, State: Initialized, Role: FOLLOWER
I20260927 10:43:44.988130 2622 ts_tablet_manager.cc:1397] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260927 10:43:44.988204 2623 consensus_queue.cc:260] T f2cb5bb17b1940f3acbd647e74b09d21 P 9fdba0a8ce2e480e8d8ecea0699bc830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:44.988436 2623 ts_tablet_manager.cc:1428] T f2cb5bb17b1940f3acbd647e74b09d21 P 9fdba0a8ce2e480e8d8ecea0699bc830: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:43:44.988586 2622 raft_consensus.cc:348] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:44.988699 2622 raft_consensus.cc:374] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:44.988730 2622 raft_consensus.cc:729] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3440908a4a4f4d7190c56635dbd7c4ca, State: Initialized, Role: FOLLOWER
I20260927 10:43:44.988842 2622 consensus_queue.cc:260] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:44.989122 2622 ts_tablet_manager.cc:1428] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:43:44.989396 2621 tablet_bootstrap.cc:492] T f2cb5bb17b1940f3acbd647e74b09d21 P 16badd963c564d45ad782203936af1ae: Bootstrap starting.
I20260927 10:43:44.990178 2621 tablet_bootstrap.cc:654] T f2cb5bb17b1940f3acbd647e74b09d21 P 16badd963c564d45ad782203936af1ae: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:44.991472 2621 tablet_bootstrap.cc:492] T f2cb5bb17b1940f3acbd647e74b09d21 P 16badd963c564d45ad782203936af1ae: No bootstrap required, opened a new log
I20260927 10:43:44.991573 2621 ts_tablet_manager.cc:1397] T f2cb5bb17b1940f3acbd647e74b09d21 P 16badd963c564d45ad782203936af1ae: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 10:43:44.991942 2621 raft_consensus.cc:348] T f2cb5bb17b1940f3acbd647e74b09d21 P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:44.992054 2621 raft_consensus.cc:374] T f2cb5bb17b1940f3acbd647e74b09d21 P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:44.992086 2621 raft_consensus.cc:729] T f2cb5bb17b1940f3acbd647e74b09d21 P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16badd963c564d45ad782203936af1ae, State: Initialized, Role: FOLLOWER
I20260927 10:43:44.992197 2621 consensus_queue.cc:260] T f2cb5bb17b1940f3acbd647e74b09d21 P 16badd963c564d45ad782203936af1ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:44.992450 2621 ts_tablet_manager.cc:1428] T f2cb5bb17b1940f3acbd647e74b09d21 P 16badd963c564d45ad782203936af1ae: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:43:45.073693 2628 raft_consensus.cc:482] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:43:45.073848 2628 raft_consensus.cc:504] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:45.074308 2628 leader_election.cc:288] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16badd963c564d45ad782203936af1ae (127.0.98.1:44899), 9fdba0a8ce2e480e8d8ecea0699bc830 (127.0.98.3:36219)
I20260927 10:43:45.074923 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2cb5bb17b1940f3acbd647e74b09d21" candidate_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" is_pre_election: true
I20260927 10:43:45.074924 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2cb5bb17b1940f3acbd647e74b09d21" candidate_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16badd963c564d45ad782203936af1ae" is_pre_election: true
I20260927 10:43:45.075073 2272 raft_consensus.cc:2454] T f2cb5bb17b1940f3acbd647e74b09d21 P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3440908a4a4f4d7190c56635dbd7c4ca in term 0.
I20260927 10:43:45.075073 2552 raft_consensus.cc:2454] T f2cb5bb17b1940f3acbd647e74b09d21 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3440908a4a4f4d7190c56635dbd7c4ca in term 0.
I20260927 10:43:45.075398 2343 leader_election.cc:302] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3440908a4a4f4d7190c56635dbd7c4ca, 9fdba0a8ce2e480e8d8ecea0699bc830; no voters:
I20260927 10:43:45.075584 2628 raft_consensus.cc:2781] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:43:45.075637 2628 raft_consensus.cc:482] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:43:45.075676 2628 raft_consensus.cc:3037] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:45.076952 2628 raft_consensus.cc:504] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:45.077343 2628 leader_election.cc:288] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca [CANDIDATE]: Term 1 election: Requested vote from peers 16badd963c564d45ad782203936af1ae (127.0.98.1:44899), 9fdba0a8ce2e480e8d8ecea0699bc830 (127.0.98.3:36219)
I20260927 10:43:45.077696 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2cb5bb17b1940f3acbd647e74b09d21" candidate_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16badd963c564d45ad782203936af1ae"
I20260927 10:43:45.077855 2272 raft_consensus.cc:3037] T f2cb5bb17b1940f3acbd647e74b09d21 P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:45.077836 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2cb5bb17b1940f3acbd647e74b09d21" candidate_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830"
I20260927 10:43:45.077965 2552 raft_consensus.cc:3037] T f2cb5bb17b1940f3acbd647e74b09d21 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:45.079087 2552 raft_consensus.cc:2454] T f2cb5bb17b1940f3acbd647e74b09d21 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3440908a4a4f4d7190c56635dbd7c4ca in term 1.
I20260927 10:43:45.079087 2272 raft_consensus.cc:2454] T f2cb5bb17b1940f3acbd647e74b09d21 P 16badd963c564d45ad782203936af1ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3440908a4a4f4d7190c56635dbd7c4ca in term 1.
I20260927 10:43:45.079419 2343 leader_election.cc:302] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16badd963c564d45ad782203936af1ae, 3440908a4a4f4d7190c56635dbd7c4ca; no voters:
I20260927 10:43:45.079591 2628 raft_consensus.cc:2781] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:43:45.079677 2628 raft_consensus.cc:686] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca [term 1 LEADER]: Becoming Leader. State: Replica: 3440908a4a4f4d7190c56635dbd7c4ca, State: Running, Role: LEADER
I20260927 10:43:45.079823 2628 consensus_queue.cc:237] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:45.081799 2113 catalog_manager.cc:5495] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 3440908a4a4f4d7190c56635dbd7c4ca (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } health_report { overall_health: UNKNOWN } } }
I20260927 10:43:45.105721 2552 raft_consensus.cc:1261] T f2cb5bb17b1940f3acbd647e74b09d21 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 1 FOLLOWER]: Refusing update from remote peer 3440908a4a4f4d7190c56635dbd7c4ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:43:45.105850 2272 raft_consensus.cc:1261] T f2cb5bb17b1940f3acbd647e74b09d21 P 16badd963c564d45ad782203936af1ae [term 1 FOLLOWER]: Refusing update from remote peer 3440908a4a4f4d7190c56635dbd7c4ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:43:45.106189 2665 consensus_queue.cc:1035] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:43:45.106333 2628 consensus_queue.cc:1035] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:43:45.109747 2650 mvcc.cc:205] Tried to move back new op lower bound from 7333911859629588480 to 7333911859528646656. Current Snapshot: MvccSnapshot[applied={T|T < 7333911859629588480}]
I20260927 10:43:45.112654 2651 mvcc.cc:205] Tried to move back new op lower bound from 7333911859629588480 to 7333911859528646656. Current Snapshot: MvccSnapshot[applied={T|T < 7333911859629588480}]
W20260927 10:43:45.122330 2650 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 10:43:46.685657 2113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:43:46.685861 2113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:43:46.688439 2113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2: Sending DeleteTablet for 3 replicas of tablet f2cb5bb17b1940f3acbd647e74b09d21
I20260927 10:43:46.689061 2251 tablet_service.cc:1514] Processing DeleteTablet for tablet f2cb5bb17b1940f3acbd647e74b09d21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:43:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49598
I20260927 10:43:46.689128 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet f2cb5bb17b1940f3acbd647e74b09d21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:43:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51652
I20260927 10:43:46.689148 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet f2cb5bb17b1940f3acbd647e74b09d21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:43:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44840
I20260927 10:43:46.689451 2704 tablet_replica.cc:330] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca: stopping tablet replica
I20260927 10:43:46.689663 2705 tablet_replica.cc:330] T f2cb5bb17b1940f3acbd647e74b09d21 P 9fdba0a8ce2e480e8d8ecea0699bc830: stopping tablet replica
I20260927 10:43:46.689829 2705 raft_consensus.cc:2229] T f2cb5bb17b1940f3acbd647e74b09d21 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:43:46.689980 2705 raft_consensus.cc:2258] T f2cb5bb17b1940f3acbd647e74b09d21 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:43:46.690209 2704 raft_consensus.cc:2229] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:43:46.690526 2704 raft_consensus.cc:2258] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:43:46.690758 2705 ts_tablet_manager.cc:1905] T f2cb5bb17b1940f3acbd647e74b09d21 P 9fdba0a8ce2e480e8d8ecea0699bc830: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:43:46.691267 2704 ts_tablet_manager.cc:1905] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:43:46.692411 2703 tablet_replica.cc:330] T f2cb5bb17b1940f3acbd647e74b09d21 P 16badd963c564d45ad782203936af1ae: stopping tablet replica
I20260927 10:43:46.692602 2703 raft_consensus.cc:2229] T f2cb5bb17b1940f3acbd647e74b09d21 P 16badd963c564d45ad782203936af1ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:43:46.692785 2703 raft_consensus.cc:2258] T f2cb5bb17b1940f3acbd647e74b09d21 P 16badd963c564d45ad782203936af1ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:43:46.693574 2703 ts_tablet_manager.cc:1905] T f2cb5bb17b1940f3acbd647e74b09d21 P 16badd963c564d45ad782203936af1ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:43:46.693926 2705 ts_tablet_manager.cc:1918] T f2cb5bb17b1940f3acbd647e74b09d21 P 9fdba0a8ce2e480e8d8ecea0699bc830: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 10:43:46.694041 2705 log.cc:1196] T f2cb5bb17b1940f3acbd647e74b09d21 P 9fdba0a8ce2e480e8d8ecea0699bc830: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/f2cb5bb17b1940f3acbd647e74b09d21
I20260927 10:43:46.694373 2705 ts_tablet_manager.cc:1939] T f2cb5bb17b1940f3acbd647e74b09d21 P 9fdba0a8ce2e480e8d8ecea0699bc830: Deleting consensus metadata
I20260927 10:43:46.695051 2704 ts_tablet_manager.cc:1918] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 10:43:46.695125 2099 catalog_manager.cc:4841] TS 9fdba0a8ce2e480e8d8ecea0699bc830 (127.0.98.3:36219): tablet f2cb5bb17b1940f3acbd647e74b09d21 (table test-table [id=77e0d955747b43f8809c62dd59aec783]) successfully deleted
I20260927 10:43:46.695187 2704 log.cc:1196] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-1/wal/wals/f2cb5bb17b1940f3acbd647e74b09d21
I20260927 10:43:46.695488 2704 ts_tablet_manager.cc:1939] T f2cb5bb17b1940f3acbd647e74b09d21 P 3440908a4a4f4d7190c56635dbd7c4ca: Deleting consensus metadata
I20260927 10:43:46.696209 2101 catalog_manager.cc:4841] TS 3440908a4a4f4d7190c56635dbd7c4ca (127.0.98.2:44795): tablet f2cb5bb17b1940f3acbd647e74b09d21 (table test-table [id=77e0d955747b43f8809c62dd59aec783]) successfully deleted
I20260927 10:43:46.697042 2703 ts_tablet_manager.cc:1918] T f2cb5bb17b1940f3acbd647e74b09d21 P 16badd963c564d45ad782203936af1ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 10:43:46.697191 2703 log.cc:1196] T f2cb5bb17b1940f3acbd647e74b09d21 P 16badd963c564d45ad782203936af1ae: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-0/wal/wals/f2cb5bb17b1940f3acbd647e74b09d21
I20260927 10:43:46.697477 2703 ts_tablet_manager.cc:1939] T f2cb5bb17b1940f3acbd647e74b09d21 P 16badd963c564d45ad782203936af1ae: Deleting consensus metadata
I20260927 10:43:46.698161 2099 catalog_manager.cc:4841] TS 16badd963c564d45ad782203936af1ae (127.0.98.1:44899): tablet f2cb5bb17b1940f3acbd647e74b09d21 (table test-table [id=77e0d955747b43f8809c62dd59aec783]) successfully deleted
Sep 27 10:43:46 dist-test-slave-dzhw krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505826, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 10:43:46 dist-test-slave-dzhw krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505826, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:46.724105 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:46.708511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41992 (local address 127.0.98.62:36115)
0927 10:43:46.708779 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:46.708791 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:46.708815 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:43:46.709028 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:46.709031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:46.709075 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:46.709171 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:46.709859 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:46.710407 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:46.711339 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:46.711574 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:46.711681 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:46.720843 (+ 9162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:46.720846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:46.720848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:46.720881 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:46.722341 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:46.722821 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:46.722825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:46.722827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:46.722909 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:46.723194 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:46.723199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:46.723201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:46.723396 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:46.723536 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:46.723788 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 10:43:46.723915 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":87,"threads_started":1}
I20260927 10:43:46.727037 2113 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:43:46.727463 2113 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:43:46.732412 2251 tablet_service.cc:1467] Processing CreateTablet for tablet 846395ba4b074f21a240bea713c6ec66 (DEFAULT_TABLE table=test-table [id=05db21ae9cee45c59fcc7b2a4d51baff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:46.732487 2392 tablet_service.cc:1467] Processing CreateTablet for tablet 846395ba4b074f21a240bea713c6ec66 (DEFAULT_TABLE table=test-table [id=05db21ae9cee45c59fcc7b2a4d51baff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:46.732628 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 846395ba4b074f21a240bea713c6ec66 (DEFAULT_TABLE table=test-table [id=05db21ae9cee45c59fcc7b2a4d51baff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:46.732730 2251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 846395ba4b074f21a240bea713c6ec66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:46.732729 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 846395ba4b074f21a240bea713c6ec66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:46.732920 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 846395ba4b074f21a240bea713c6ec66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:46.735365 2714 tablet_bootstrap.cc:492] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830: Bootstrap starting.
I20260927 10:43:46.735497 2715 tablet_bootstrap.cc:492] T 846395ba4b074f21a240bea713c6ec66 P 16badd963c564d45ad782203936af1ae: Bootstrap starting.
I20260927 10:43:46.735739 2716 tablet_bootstrap.cc:492] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca: Bootstrap starting.
I20260927 10:43:46.736354 2715 tablet_bootstrap.cc:654] T 846395ba4b074f21a240bea713c6ec66 P 16badd963c564d45ad782203936af1ae: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:46.736411 2714 tablet_bootstrap.cc:654] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:46.736729 2716 tablet_bootstrap.cc:654] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:46.737582 2715 tablet_bootstrap.cc:492] T 846395ba4b074f21a240bea713c6ec66 P 16badd963c564d45ad782203936af1ae: No bootstrap required, opened a new log
I20260927 10:43:46.737670 2715 ts_tablet_manager.cc:1397] T 846395ba4b074f21a240bea713c6ec66 P 16badd963c564d45ad782203936af1ae: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 10:43:46.737658 2714 tablet_bootstrap.cc:492] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830: No bootstrap required, opened a new log
I20260927 10:43:46.737758 2714 ts_tablet_manager.cc:1397] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 10:43:46.738250 2714 raft_consensus.cc:348] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:46.738293 2715 raft_consensus.cc:348] T 846395ba4b074f21a240bea713c6ec66 P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:46.738375 2714 raft_consensus.cc:374] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:46.738442 2715 raft_consensus.cc:374] T 846395ba4b074f21a240bea713c6ec66 P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:46.738463 2714 raft_consensus.cc:729] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fdba0a8ce2e480e8d8ecea0699bc830, State: Initialized, Role: FOLLOWER
I20260927 10:43:46.738492 2715 raft_consensus.cc:729] T 846395ba4b074f21a240bea713c6ec66 P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16badd963c564d45ad782203936af1ae, State: Initialized, Role: FOLLOWER
I20260927 10:43:46.738631 2715 consensus_queue.cc:260] T 846395ba4b074f21a240bea713c6ec66 P 16badd963c564d45ad782203936af1ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:46.738922 2715 ts_tablet_manager.cc:1428] T 846395ba4b074f21a240bea713c6ec66 P 16badd963c564d45ad782203936af1ae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:43:46.738976 2716 tablet_bootstrap.cc:492] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca: No bootstrap required, opened a new log
I20260927 10:43:46.739070 2716 ts_tablet_manager.cc:1397] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:43:46.739445 2716 raft_consensus.cc:348] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:46.739537 2716 raft_consensus.cc:374] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:46.739558 2716 raft_consensus.cc:729] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3440908a4a4f4d7190c56635dbd7c4ca, State: Initialized, Role: FOLLOWER
I20260927 10:43:46.739701 2716 consensus_queue.cc:260] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:46.739979 2716 ts_tablet_manager.cc:1428] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:43:46.740111 2714 consensus_queue.cc:260] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:46.740497 2714 ts_tablet_manager.cc:1428] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260927 10:43:46.765987 2720 raft_consensus.cc:482] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:43:46.766160 2720 raft_consensus.cc:504] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:46.767338 2720 leader_election.cc:288] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16badd963c564d45ad782203936af1ae (127.0.98.1:44899), 3440908a4a4f4d7190c56635dbd7c4ca (127.0.98.2:44795)
I20260927 10:43:46.771023 2697 raft_consensus.cc:482] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:43:46.771147 2697 raft_consensus.cc:504] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:46.771577 2697 leader_election.cc:288] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16badd963c564d45ad782203936af1ae (127.0.98.1:44899), 9fdba0a8ce2e480e8d8ecea0699bc830 (127.0.98.3:36219)
I20260927 10:43:46.771914 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846395ba4b074f21a240bea713c6ec66" candidate_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16badd963c564d45ad782203936af1ae" is_pre_election: true
I20260927 10:43:46.772023 2721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:46.767580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44795 (local address 127.0.98.3:45917)
0927 10:43:46.767925 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:46.767943 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:46.768068 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:46.768392 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:46.768395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:46.768415 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:43:46.768570 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:46.768577 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:46.769367 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:46.769370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:46.770604 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:46.770613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:46.771539 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:46.771542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:46.771652 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:46.771680 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 10:43:46.771784 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 10:43:46.771855 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":72,"thread_start_us":137,"threads_started":1}
I20260927 10:43:46.771976 2551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846395ba4b074f21a240bea713c6ec66" candidate_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" is_pre_election: true
I20260927 10:43:46.772055 2272 raft_consensus.cc:2454] T 846395ba4b074f21a240bea713c6ec66 P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3440908a4a4f4d7190c56635dbd7c4ca in term 0.
I20260927 10:43:46.772130 2551 raft_consensus.cc:2454] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3440908a4a4f4d7190c56635dbd7c4ca in term 0.
I20260927 10:43:46.772365 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:46.767676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45917 (local address 127.0.98.2:44795)
0927 10:43:46.767956 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:46.767964 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:46.768014 (+ 50us) server_negotiation.cc:407] Connection header received
0927 10:43:46.768198 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:46.768202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:46.768268 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:46.768357 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:43:46.768683 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:46.769263 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:46.770736 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:46.771312 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:46.771415 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:46.771493 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:46.772161 (+ 668us) server_negotiation.cc:299] Negotiation successful
0927 10:43:46.772239 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":101,"threads_started":1}
I20260927 10:43:46.772364 2343 leader_election.cc:302] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16badd963c564d45ad782203936af1ae, 3440908a4a4f4d7190c56635dbd7c4ca; no voters:
I20260927 10:43:46.772535 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:46.767668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40865 (local address 127.0.98.1:44899)
0927 10:43:46.768015 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:46.768025 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:46.768338 (+ 313us) server_negotiation.cc:407] Connection header received
0927 10:43:46.768398 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:46.768401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:46.768446 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:46.768535 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:43:46.768846 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:46.769475 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:46.770715 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:46.771273 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:46.771391 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:46.771462 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:46.771953 (+ 491us) server_negotiation.cc:299] Negotiation successful
0927 10:43:46.772032 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":143,"threads_started":1}
I20260927 10:43:46.772830 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846395ba4b074f21a240bea713c6ec66" candidate_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" is_pre_election: true
I20260927 10:43:46.772938 2412 raft_consensus.cc:2454] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fdba0a8ce2e480e8d8ecea0699bc830 in term 0.
I20260927 10:43:46.772998 2697 raft_consensus.cc:2781] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:43:46.773053 2697 raft_consensus.cc:482] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:43:46.773099 2697 raft_consensus.cc:3037] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:46.773262 2485 leader_election.cc:302] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3440908a4a4f4d7190c56635dbd7c4ca, 9fdba0a8ce2e480e8d8ecea0699bc830; no voters:
I20260927 10:43:46.773521 2720 raft_consensus.cc:2781] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:43:46.773581 2720 raft_consensus.cc:482] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:43:46.773641 2720 raft_consensus.cc:3037] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:46.773753 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:46.767582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44899 (local address 127.0.98.3:40865)
0927 10:43:46.768200 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:46.768215 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:46.768318 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:46.768548 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:46.768550 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:46.768565 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:43:46.768732 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:46.768739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:46.769615 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:46.769617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:46.770601 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:46.770610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:46.771398 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:46.771402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:46.771503 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:46.771826 (+ 323us) client_negotiation.cc:769] Sending connection context
0927 10:43:46.773461 (+ 1635us) client_negotiation.cc:240] Negotiation successful
0927 10:43:46.773544 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":336,"thread_start_us":106,"threads_started":1}
I20260927 10:43:46.774153 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846395ba4b074f21a240bea713c6ec66" candidate_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16badd963c564d45ad782203936af1ae" is_pre_election: true
I20260927 10:43:46.774281 2272 raft_consensus.cc:2454] T 846395ba4b074f21a240bea713c6ec66 P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fdba0a8ce2e480e8d8ecea0699bc830 in term 0.
I20260927 10:43:46.774379 2697 raft_consensus.cc:504] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:46.774746 2697 leader_election.cc:288] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca [CANDIDATE]: Term 1 election: Requested vote from peers 16badd963c564d45ad782203936af1ae (127.0.98.1:44899), 9fdba0a8ce2e480e8d8ecea0699bc830 (127.0.98.3:36219)
I20260927 10:43:46.774785 2720 raft_consensus.cc:504] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:46.775048 2551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846395ba4b074f21a240bea713c6ec66" candidate_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830"
I20260927 10:43:46.775092 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846395ba4b074f21a240bea713c6ec66" candidate_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16badd963c564d45ad782203936af1ae"
I20260927 10:43:46.775126 2720 leader_election.cc:288] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830 [CANDIDATE]: Term 1 election: Requested vote from peers 16badd963c564d45ad782203936af1ae (127.0.98.1:44899), 3440908a4a4f4d7190c56635dbd7c4ca (127.0.98.2:44795)
I20260927 10:43:46.775192 2272 raft_consensus.cc:3037] T 846395ba4b074f21a240bea713c6ec66 P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:46.775223 2551 raft_consensus.cc:2379] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3440908a4a4f4d7190c56635dbd7c4ca in current term 1: Already voted for candidate 9fdba0a8ce2e480e8d8ecea0699bc830 in this term.
I20260927 10:43:46.775522 2271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846395ba4b074f21a240bea713c6ec66" candidate_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16badd963c564d45ad782203936af1ae"
I20260927 10:43:46.775637 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846395ba4b074f21a240bea713c6ec66" candidate_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3440908a4a4f4d7190c56635dbd7c4ca"
I20260927 10:43:46.775770 2412 raft_consensus.cc:2379] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9fdba0a8ce2e480e8d8ecea0699bc830 in current term 1: Already voted for candidate 3440908a4a4f4d7190c56635dbd7c4ca in this term.
I20260927 10:43:46.776152 2485 leader_election.cc:302] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9fdba0a8ce2e480e8d8ecea0699bc830; no voters: 16badd963c564d45ad782203936af1ae, 3440908a4a4f4d7190c56635dbd7c4ca
I20260927 10:43:46.776317 2720 raft_consensus.cc:2726] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 10:43:46.776384 2272 raft_consensus.cc:2454] T 846395ba4b074f21a240bea713c6ec66 P 16badd963c564d45ad782203936af1ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3440908a4a4f4d7190c56635dbd7c4ca in term 1.
I20260927 10:43:46.776633 2343 leader_election.cc:302] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16badd963c564d45ad782203936af1ae, 3440908a4a4f4d7190c56635dbd7c4ca; no voters: 9fdba0a8ce2e480e8d8ecea0699bc830
I20260927 10:43:46.776815 2697 raft_consensus.cc:2781] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:43:46.776928 2697 raft_consensus.cc:686] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca [term 1 LEADER]: Becoming Leader. State: Replica: 3440908a4a4f4d7190c56635dbd7c4ca, State: Running, Role: LEADER
I20260927 10:43:46.777063 2697 consensus_queue.cc:237] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:46.778687 2113 catalog_manager.cc:5495] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 3440908a4a4f4d7190c56635dbd7c4ca (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } health_report { overall_health: UNKNOWN } } }
I20260927 10:43:46.797217 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:46.794302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51676 (local address 127.0.98.2:44795)
0927 10:43:46.794464 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:46.794471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:46.794485 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:43:46.794597 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:46.794599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:46.794641 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:46.794705 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:43:46.794900 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:46.795444 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:46.796260 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:46.796488 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:46.796589 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:46.796833 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:43:46.796906 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:43:46.796972 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:46.797059 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 10:43:46.797106 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 10:43:46.803277 2551 raft_consensus.cc:1261] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 1 FOLLOWER]: Refusing update from remote peer 3440908a4a4f4d7190c56635dbd7c4ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:43:46.803272 2272 raft_consensus.cc:1261] T 846395ba4b074f21a240bea713c6ec66 P 16badd963c564d45ad782203936af1ae [term 1 FOLLOWER]: Refusing update from remote peer 3440908a4a4f4d7190c56635dbd7c4ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:43:46.803622 2693 consensus_queue.cc:1035] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:43:46.803784 2697 consensus_queue.cc:1035] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:43:46.834836 2113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:43:46.834983 2113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:43:46.837062 2113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2: Sending DeleteTablet for 3 replicas of tablet 846395ba4b074f21a240bea713c6ec66
I20260927 10:43:46.837605 2251 tablet_service.cc:1514] Processing DeleteTablet for tablet 846395ba4b074f21a240bea713c6ec66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:43:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49598
I20260927 10:43:46.837711 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet 846395ba4b074f21a240bea713c6ec66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:43:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44840
I20260927 10:43:46.837813 2703 tablet_replica.cc:330] T 846395ba4b074f21a240bea713c6ec66 P 16badd963c564d45ad782203936af1ae: stopping tablet replica
I20260927 10:43:46.837797 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet 846395ba4b074f21a240bea713c6ec66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:43:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51652
I20260927 10:43:46.837910 2705 tablet_replica.cc:330] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830: stopping tablet replica
I20260927 10:43:46.837927 2703 raft_consensus.cc:2229] T 846395ba4b074f21a240bea713c6ec66 P 16badd963c564d45ad782203936af1ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:43:46.837955 2704 tablet_replica.cc:330] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca: stopping tablet replica
I20260927 10:43:46.838011 2705 raft_consensus.cc:2229] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:43:46.838040 2703 raft_consensus.cc:2258] T 846395ba4b074f21a240bea713c6ec66 P 16badd963c564d45ad782203936af1ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:43:46.838065 2704 raft_consensus.cc:2229] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:43:46.838137 2705 raft_consensus.cc:2258] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:43:46.838387 2704 raft_consensus.cc:2258] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:43:46.838660 2113 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 10:43:46.838915 2703 ts_tablet_manager.cc:1905] T 846395ba4b074f21a240bea713c6ec66 P 16badd963c564d45ad782203936af1ae: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 10:43:46.839099 2113 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:43:46.839293 2704 ts_tablet_manager.cc:1905] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:43:46.841590 2703 ts_tablet_manager.cc:1918] T 846395ba4b074f21a240bea713c6ec66 P 16badd963c564d45ad782203936af1ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:43:46.841678 2703 log.cc:1196] T 846395ba4b074f21a240bea713c6ec66 P 16badd963c564d45ad782203936af1ae: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-0/wal/wals/846395ba4b074f21a240bea713c6ec66
I20260927 10:43:46.841719 2704 ts_tablet_manager.cc:1918] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:43:46.841790 2704 log.cc:1196] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-1/wal/wals/846395ba4b074f21a240bea713c6ec66
I20260927 10:43:46.841951 2703 ts_tablet_manager.cc:1939] T 846395ba4b074f21a240bea713c6ec66 P 16badd963c564d45ad782203936af1ae: Deleting consensus metadata
I20260927 10:43:46.842054 2704 ts_tablet_manager.cc:1939] T 846395ba4b074f21a240bea713c6ec66 P 3440908a4a4f4d7190c56635dbd7c4ca: Deleting consensus metadata
I20260927 10:43:46.842628 2101 catalog_manager.cc:4841] TS 3440908a4a4f4d7190c56635dbd7c4ca (127.0.98.2:44795): tablet 846395ba4b074f21a240bea713c6ec66 (table test-table [id=05db21ae9cee45c59fcc7b2a4d51baff]) successfully deleted
I20260927 10:43:46.842859 2099 catalog_manager.cc:4841] TS 16badd963c564d45ad782203936af1ae (127.0.98.1:44899): tablet 846395ba4b074f21a240bea713c6ec66 (table test-table [id=05db21ae9cee45c59fcc7b2a4d51baff]) successfully deleted
I20260927 10:43:46.843528 2705 ts_tablet_manager.cc:1905] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:43:46.843581 2392 tablet_service.cc:1467] Processing CreateTablet for tablet 3168d203a6534b35899670c3b94aaef7 (DEFAULT_TABLE table=test-table [id=bf34d1b766794b9d87b5e7fcd77c16f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:46.843760 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3168d203a6534b35899670c3b94aaef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:46.843757 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 3168d203a6534b35899670c3b94aaef7 (DEFAULT_TABLE table=test-table [id=bf34d1b766794b9d87b5e7fcd77c16f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:46.843981 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3168d203a6534b35899670c3b94aaef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:46.845474 2716 tablet_bootstrap.cc:492] T 3168d203a6534b35899670c3b94aaef7 P 3440908a4a4f4d7190c56635dbd7c4ca: Bootstrap starting.
I20260927 10:43:46.845901 2705 ts_tablet_manager.cc:1918] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:43:46.845969 2705 log.cc:1196] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/846395ba4b074f21a240bea713c6ec66
I20260927 10:43:46.845996 2714 tablet_bootstrap.cc:492] T 3168d203a6534b35899670c3b94aaef7 P 9fdba0a8ce2e480e8d8ecea0699bc830: Bootstrap starting.
I20260927 10:43:46.846251 2705 ts_tablet_manager.cc:1939] T 846395ba4b074f21a240bea713c6ec66 P 9fdba0a8ce2e480e8d8ecea0699bc830: Deleting consensus metadata
I20260927 10:43:46.846280 2716 tablet_bootstrap.cc:654] T 3168d203a6534b35899670c3b94aaef7 P 3440908a4a4f4d7190c56635dbd7c4ca: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:46.846885 2099 catalog_manager.cc:4841] TS 9fdba0a8ce2e480e8d8ecea0699bc830 (127.0.98.3:36219): tablet 846395ba4b074f21a240bea713c6ec66 (table test-table [id=05db21ae9cee45c59fcc7b2a4d51baff]) successfully deleted
I20260927 10:43:46.846980 2714 tablet_bootstrap.cc:654] T 3168d203a6534b35899670c3b94aaef7 P 9fdba0a8ce2e480e8d8ecea0699bc830: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:46.847138 2251 tablet_service.cc:1467] Processing CreateTablet for tablet 3168d203a6534b35899670c3b94aaef7 (DEFAULT_TABLE table=test-table [id=bf34d1b766794b9d87b5e7fcd77c16f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:46.847409 2251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3168d203a6534b35899670c3b94aaef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:46.847501 2716 tablet_bootstrap.cc:492] T 3168d203a6534b35899670c3b94aaef7 P 3440908a4a4f4d7190c56635dbd7c4ca: No bootstrap required, opened a new log
I20260927 10:43:46.847573 2716 ts_tablet_manager.cc:1397] T 3168d203a6534b35899670c3b94aaef7 P 3440908a4a4f4d7190c56635dbd7c4ca: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 10:43:46.848028 2716 raft_consensus.cc:348] T 3168d203a6534b35899670c3b94aaef7 P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:46.848129 2716 raft_consensus.cc:374] T 3168d203a6534b35899670c3b94aaef7 P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:46.848157 2716 raft_consensus.cc:729] T 3168d203a6534b35899670c3b94aaef7 P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3440908a4a4f4d7190c56635dbd7c4ca, State: Initialized, Role: FOLLOWER
I20260927 10:43:46.848274 2716 consensus_queue.cc:260] T 3168d203a6534b35899670c3b94aaef7 P 3440908a4a4f4d7190c56635dbd7c4ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:46.848403 2714 tablet_bootstrap.cc:492] T 3168d203a6534b35899670c3b94aaef7 P 9fdba0a8ce2e480e8d8ecea0699bc830: No bootstrap required, opened a new log
I20260927 10:43:46.848461 2714 ts_tablet_manager.cc:1397] T 3168d203a6534b35899670c3b94aaef7 P 9fdba0a8ce2e480e8d8ecea0699bc830: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260927 10:43:46.848546 2716 ts_tablet_manager.cc:1428] T 3168d203a6534b35899670c3b94aaef7 P 3440908a4a4f4d7190c56635dbd7c4ca: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 10:43:46.848824 2714 raft_consensus.cc:348] T 3168d203a6534b35899670c3b94aaef7 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:46.848930 2714 raft_consensus.cc:374] T 3168d203a6534b35899670c3b94aaef7 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:46.848973 2714 raft_consensus.cc:729] T 3168d203a6534b35899670c3b94aaef7 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fdba0a8ce2e480e8d8ecea0699bc830, State: Initialized, Role: FOLLOWER
I20260927 10:43:46.849097 2714 consensus_queue.cc:260] T 3168d203a6534b35899670c3b94aaef7 P 9fdba0a8ce2e480e8d8ecea0699bc830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:46.849992 2715 tablet_bootstrap.cc:492] T 3168d203a6534b35899670c3b94aaef7 P 16badd963c564d45ad782203936af1ae: Bootstrap starting.
I20260927 10:43:46.850732 2714 ts_tablet_manager.cc:1428] T 3168d203a6534b35899670c3b94aaef7 P 9fdba0a8ce2e480e8d8ecea0699bc830: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 10:43:46.850848 2715 tablet_bootstrap.cc:654] T 3168d203a6534b35899670c3b94aaef7 P 16badd963c564d45ad782203936af1ae: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:46.852043 2715 tablet_bootstrap.cc:492] T 3168d203a6534b35899670c3b94aaef7 P 16badd963c564d45ad782203936af1ae: No bootstrap required, opened a new log
I20260927 10:43:46.852129 2715 ts_tablet_manager.cc:1397] T 3168d203a6534b35899670c3b94aaef7 P 16badd963c564d45ad782203936af1ae: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:43:46.852512 2715 raft_consensus.cc:348] T 3168d203a6534b35899670c3b94aaef7 P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:46.852622 2715 raft_consensus.cc:374] T 3168d203a6534b35899670c3b94aaef7 P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:46.852665 2715 raft_consensus.cc:729] T 3168d203a6534b35899670c3b94aaef7 P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16badd963c564d45ad782203936af1ae, State: Initialized, Role: FOLLOWER
I20260927 10:43:46.852792 2715 consensus_queue.cc:260] T 3168d203a6534b35899670c3b94aaef7 P 16badd963c564d45ad782203936af1ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:46.853057 2715 ts_tablet_manager.cc:1428] T 3168d203a6534b35899670c3b94aaef7 P 16badd963c564d45ad782203936af1ae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:43:47.081193 2680 raft_consensus.cc:482] T 3168d203a6534b35899670c3b94aaef7 P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:43:47.081370 2680 raft_consensus.cc:504] T 3168d203a6534b35899670c3b94aaef7 P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:47.081871 2680 leader_election.cc:288] T 3168d203a6534b35899670c3b94aaef7 P 16badd963c564d45ad782203936af1ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3440908a4a4f4d7190c56635dbd7c4ca (127.0.98.2:44795), 9fdba0a8ce2e480e8d8ecea0699bc830 (127.0.98.3:36219)
I20260927 10:43:47.082355 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3168d203a6534b35899670c3b94aaef7" candidate_uuid: "16badd963c564d45ad782203936af1ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" is_pre_election: true
I20260927 10:43:47.082422 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3168d203a6534b35899670c3b94aaef7" candidate_uuid: "16badd963c564d45ad782203936af1ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" is_pre_election: true
I20260927 10:43:47.082583 2552 raft_consensus.cc:2454] T 3168d203a6534b35899670c3b94aaef7 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16badd963c564d45ad782203936af1ae in term 0.
I20260927 10:43:47.082580 2412 raft_consensus.cc:2454] T 3168d203a6534b35899670c3b94aaef7 P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16badd963c564d45ad782203936af1ae in term 0.
I20260927 10:43:47.082917 2205 leader_election.cc:302] T 3168d203a6534b35899670c3b94aaef7 P 16badd963c564d45ad782203936af1ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16badd963c564d45ad782203936af1ae, 3440908a4a4f4d7190c56635dbd7c4ca; no voters:
I20260927 10:43:47.083097 2680 raft_consensus.cc:2781] T 3168d203a6534b35899670c3b94aaef7 P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:43:47.083155 2680 raft_consensus.cc:482] T 3168d203a6534b35899670c3b94aaef7 P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:43:47.083204 2680 raft_consensus.cc:3037] T 3168d203a6534b35899670c3b94aaef7 P 16badd963c564d45ad782203936af1ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:47.084174 2680 raft_consensus.cc:504] T 3168d203a6534b35899670c3b94aaef7 P 16badd963c564d45ad782203936af1ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:47.084515 2680 leader_election.cc:288] T 3168d203a6534b35899670c3b94aaef7 P 16badd963c564d45ad782203936af1ae [CANDIDATE]: Term 1 election: Requested vote from peers 3440908a4a4f4d7190c56635dbd7c4ca (127.0.98.2:44795), 9fdba0a8ce2e480e8d8ecea0699bc830 (127.0.98.3:36219)
I20260927 10:43:47.084841 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3168d203a6534b35899670c3b94aaef7" candidate_uuid: "16badd963c564d45ad782203936af1ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3440908a4a4f4d7190c56635dbd7c4ca"
I20260927 10:43:47.084935 2412 raft_consensus.cc:3037] T 3168d203a6534b35899670c3b94aaef7 P 3440908a4a4f4d7190c56635dbd7c4ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:47.084947 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3168d203a6534b35899670c3b94aaef7" candidate_uuid: "16badd963c564d45ad782203936af1ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830"
I20260927 10:43:47.085081 2552 raft_consensus.cc:3037] T 3168d203a6534b35899670c3b94aaef7 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:47.085892 2412 raft_consensus.cc:2454] T 3168d203a6534b35899670c3b94aaef7 P 3440908a4a4f4d7190c56635dbd7c4ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16badd963c564d45ad782203936af1ae in term 1.
I20260927 10:43:47.086046 2552 raft_consensus.cc:2454] T 3168d203a6534b35899670c3b94aaef7 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16badd963c564d45ad782203936af1ae in term 1.
I20260927 10:43:47.086211 2205 leader_election.cc:302] T 3168d203a6534b35899670c3b94aaef7 P 16badd963c564d45ad782203936af1ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16badd963c564d45ad782203936af1ae, 3440908a4a4f4d7190c56635dbd7c4ca; no voters:
I20260927 10:43:47.086391 2680 raft_consensus.cc:2781] T 3168d203a6534b35899670c3b94aaef7 P 16badd963c564d45ad782203936af1ae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:43:47.086516 2680 raft_consensus.cc:686] T 3168d203a6534b35899670c3b94aaef7 P 16badd963c564d45ad782203936af1ae [term 1 LEADER]: Becoming Leader. State: Replica: 16badd963c564d45ad782203936af1ae, State: Running, Role: LEADER
I20260927 10:43:47.086661 2680 consensus_queue.cc:237] T 3168d203a6534b35899670c3b94aaef7 P 16badd963c564d45ad782203936af1ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } }
I20260927 10:43:47.088562 2113 catalog_manager.cc:5495] T 3168d203a6534b35899670c3b94aaef7 P 16badd963c564d45ad782203936af1ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 16badd963c564d45ad782203936af1ae (127.0.98.1). New cstate: current_term: 1 leader_uuid: "16badd963c564d45ad782203936af1ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16badd963c564d45ad782203936af1ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 } health_report { overall_health: UNKNOWN } } }
I20260927 10:43:47.151892 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:47.148976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49618 (local address 127.0.98.1:44899)
0927 10:43:47.149074 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:47.149077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:47.149094 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:43:47.149235 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:47.149237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:47.149278 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:47.149337 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:43:47.149550 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:47.150064 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:47.150954 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:47.151156 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:47.151235 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:47.151468 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:43:47.151532 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:43:47.151584 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:47.151669 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 10:43:47.151711 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 10:43:47.160653 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:47.157758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44361 (local address 127.0.98.1:44899)
0927 10:43:47.157911 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:47.157915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:47.157999 (+ 84us) server_negotiation.cc:407] Connection header received
0927 10:43:47.158089 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:47.158091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:47.158131 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:47.158195 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:43:47.158385 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:47.158912 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:47.159676 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:47.159883 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:47.159958 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:47.160246 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:43:47.160307 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:43:47.160362 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:47.160498 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 10:43:47.160542 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 10:43:47.160653 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:47.157659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44899 (local address 127.0.98.1:44361)
0927 10:43:47.157912 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:47.157930 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:47.158015 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:47.158217 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:47.158219 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:47.158225 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:43:47.158303 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:47.158307 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:47.158998 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:47.159000 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:47.159574 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:47.159582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:47.160012 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:47.160018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:47.160113 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:47.160143 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:43:47.160380 (+ 237us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:43:47.160385 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 10:43:47.160441 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 10:43:47.160502 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":97,"threads_started":1}
I20260927 10:43:47.166173 2552 raft_consensus.cc:1261] T 3168d203a6534b35899670c3b94aaef7 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 1 FOLLOWER]: Refusing update from remote peer 16badd963c564d45ad782203936af1ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:43:47.166244 2412 raft_consensus.cc:1261] T 3168d203a6534b35899670c3b94aaef7 P 3440908a4a4f4d7190c56635dbd7c4ca [term 1 FOLLOWER]: Refusing update from remote peer 16badd963c564d45ad782203936af1ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:43:47.166637 2683 consensus_queue.cc:1035] T 3168d203a6534b35899670c3b94aaef7 P 16badd963c564d45ad782203936af1ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "3440908a4a4f4d7190c56635dbd7c4ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:43:47.166849 2728 consensus_queue.cc:1035] T 3168d203a6534b35899670c3b94aaef7 P 16badd963c564d45ad782203936af1ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fdba0a8ce2e480e8d8ecea0699bc830" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 10:43:47.181630 2681 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 10:43:48.735076 2113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:43:48.735262 2113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:43:48.737798 2113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe9200f9a065422fa6ad6eccded6f5d2: Sending DeleteTablet for 3 replicas of tablet 3168d203a6534b35899670c3b94aaef7
I20260927 10:43:48.738461 2251 tablet_service.cc:1514] Processing DeleteTablet for tablet 3168d203a6534b35899670c3b94aaef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:43:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49598
I20260927 10:43:48.738485 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet 3168d203a6534b35899670c3b94aaef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:43:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51652
I20260927 10:43:48.738533 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet 3168d203a6534b35899670c3b94aaef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:43:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44840
I20260927 10:43:48.738837 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 2187
I20260927 10:43:48.739684 2763 tablet_replica.cc:330] T 3168d203a6534b35899670c3b94aaef7 P 9fdba0a8ce2e480e8d8ecea0699bc830: stopping tablet replica
I20260927 10:43:48.739857 2763 raft_consensus.cc:2229] T 3168d203a6534b35899670c3b94aaef7 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:43:48.739990 2763 raft_consensus.cc:2258] T 3168d203a6534b35899670c3b94aaef7 P 9fdba0a8ce2e480e8d8ecea0699bc830 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:43:48.740521 2762 tablet_replica.cc:330] T 3168d203a6534b35899670c3b94aaef7 P 3440908a4a4f4d7190c56635dbd7c4ca: stopping tablet replica
I20260927 10:43:48.740648 2762 raft_consensus.cc:2229] T 3168d203a6534b35899670c3b94aaef7 P 3440908a4a4f4d7190c56635dbd7c4ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:43:48.740662 2763 ts_tablet_manager.cc:1905] T 3168d203a6534b35899670c3b94aaef7 P 9fdba0a8ce2e480e8d8ecea0699bc830: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:43:48.740768 2762 raft_consensus.cc:2258] T 3168d203a6534b35899670c3b94aaef7 P 3440908a4a4f4d7190c56635dbd7c4ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:43:48.741335 2762 ts_tablet_manager.cc:1905] T 3168d203a6534b35899670c3b94aaef7 P 3440908a4a4f4d7190c56635dbd7c4ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:43:48.743327 2763 ts_tablet_manager.cc:1918] T 3168d203a6534b35899670c3b94aaef7 P 9fdba0a8ce2e480e8d8ecea0699bc830: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 10:43:48.743415 2763 log.cc:1196] T 3168d203a6534b35899670c3b94aaef7 P 9fdba0a8ce2e480e8d8ecea0699bc830: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/3168d203a6534b35899670c3b94aaef7
I20260927 10:43:48.743732 2763 ts_tablet_manager.cc:1939] T 3168d203a6534b35899670c3b94aaef7 P 9fdba0a8ce2e480e8d8ecea0699bc830: Deleting consensus metadata
I20260927 10:43:48.744032 2762 ts_tablet_manager.cc:1918] T 3168d203a6534b35899670c3b94aaef7 P 3440908a4a4f4d7190c56635dbd7c4ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 10:43:48.744108 2762 log.cc:1196] T 3168d203a6534b35899670c3b94aaef7 P 3440908a4a4f4d7190c56635dbd7c4ca: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505812134190-392-0/minicluster-data/ts-1/wal/wals/3168d203a6534b35899670c3b94aaef7
I20260927 10:43:48.744393 2762 ts_tablet_manager.cc:1939] T 3168d203a6534b35899670c3b94aaef7 P 3440908a4a4f4d7190c56635dbd7c4ca: Deleting consensus metadata
I20260927 10:43:48.744627 2099 catalog_manager.cc:4841] TS 9fdba0a8ce2e480e8d8ecea0699bc830 (127.0.98.3:36219): tablet 3168d203a6534b35899670c3b94aaef7 (table test-table [id=bf34d1b766794b9d87b5e7fcd77c16f1]) successfully deleted
I20260927 10:43:48.745114 2101 catalog_manager.cc:4841] TS 3440908a4a4f4d7190c56635dbd7c4ca (127.0.98.2:44795): tablet 3168d203a6534b35899670c3b94aaef7 (table test-table [id=bf34d1b766794b9d87b5e7fcd77c16f1]) successfully deleted
W20260927 10:43:48.747684 2099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44899 (error 108)
I20260927 10:43:48.748063 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 2327
I20260927 10:43:48.750448 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:48.749935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44899 (local address 127.0.0.1:49628)
0927 10:43:48.750201 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:48.750287 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
W20260927 10:43:48.750590 2099 catalog_manager.cc:4584] TS 16badd963c564d45ad782203936af1ae (127.0.98.1:44899): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3168d203a6534b35899670c3b94aaef7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44899: connect: Connection refused (error 111)
I20260927 10:43:48.754654 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 2467
I20260927 10:43:48.760406 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 2083
2026-09-27T10:43:48Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6618 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:43:48 dist-test-slave-dzhw krb5kdc[2768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:43:48 dist-test-slave-dzhw krb5kdc[2768](info): set up 2 sockets
Sep 27 10:43:48 dist-test-slave-dzhw krb5kdc[2768](info): commencing operation
krb5kdc: starting...
W20260927 10:43:49.810635 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:43:49 dist-test-slave-dzhw krb5kdc[2768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:43:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:43:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:43:49.933941 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44901
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:50.044314 2781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:50.044665 2781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:50.044749 2781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:50.049072 2781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:43:50.049178 2781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:50.049216 2781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:50.049254 2781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:43:50.049289 2781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:43:50.054843 2781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:50.055985 2781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:50.057065 2781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:50.065672 2789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:50.065702 2786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:50.065770 2781 server_base.cc:1034] running on GCE node
W20260927 10:43:50.065718 2787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:50.066517 2781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:50.067648 2781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:50.068843 2781 hybrid_clock.cc:648] HybridClock initialized: now 1790505830068827 us; error 37 us; skew 500 ppm
Sep 27 10:43:50 dist-test-slave-dzhw krb5kdc[2768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:50.071794 2781 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:43:50.072690 2781 webserver.cc:460] Webserver started at http://127.0.98.62:36183/ using document root <none> and password file <none>
I20260927 10:43:50.073258 2781 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:50.073349 2781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:50.073589 2781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:50.075706 2781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "ac4152d861cc4449b8c77a0dce86a16a"
format_stamp: "Formatted at 2026-09-27 10:43:50 on dist-test-slave-dzhw"
server_key: "3ff5560afcb7657fe2b2e91ab52a8e9f"
server_key_iv: "7b0e7eda528fcb0e4fa439e0a469ff3f"
server_key_version: "encryptionkey@0"
I20260927 10:43:50.076270 2781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "ac4152d861cc4449b8c77a0dce86a16a"
format_stamp: "Formatted at 2026-09-27 10:43:50 on dist-test-slave-dzhw"
server_key: "3ff5560afcb7657fe2b2e91ab52a8e9f"
server_key_iv: "7b0e7eda528fcb0e4fa439e0a469ff3f"
server_key_version: "encryptionkey@0"
I20260927 10:43:50.079999 2781 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:43:50.082472 2796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:50.083734 2781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:43:50.083878 2781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "ac4152d861cc4449b8c77a0dce86a16a"
format_stamp: "Formatted at 2026-09-27 10:43:50 on dist-test-slave-dzhw"
server_key: "3ff5560afcb7657fe2b2e91ab52a8e9f"
server_key_iv: "7b0e7eda528fcb0e4fa439e0a469ff3f"
server_key_version: "encryptionkey@0"
I20260927 10:43:50.083998 2781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:50.099221 2781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:50.102384 2781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:50.102664 2781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:50.109759 2781 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44901
I20260927 10:43:50.109792 2848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44901 every 8 connection(s)
I20260927 10:43:50.110798 2781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:43:50.113507 2849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:50.119570 2849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a: Bootstrap starting.
I20260927 10:43:50.120613 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 2781
I20260927 10:43:50.120707 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:43:50.120965 392 external_mini_cluster.cc:1428] Setting key 15df7c20d69d4f55c898c3309f00a4b5
I20260927 10:43:50.122351 2849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:50.123100 2849 log.cc:824] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:43:50.125231 2849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a: No bootstrap required, opened a new log
I20260927 10:43:50.129787 2849 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4152d861cc4449b8c77a0dce86a16a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44901 } }
I20260927 10:43:50.130049 2849 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:50.130129 2849 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac4152d861cc4449b8c77a0dce86a16a, State: Initialized, Role: FOLLOWER
Sep 27 10:43:50 dist-test-slave-dzhw krb5kdc[2768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:50.130890 2849 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4152d861cc4449b8c77a0dce86a16a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44901 } }
I20260927 10:43:50.131093 2849 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:43:50.131156 2854 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:43:50.131316 2854 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4152d861cc4449b8c77a0dce86a16a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44901 } }
I20260927 10:43:50.131700 2849 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:43:50.131834 2849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:50.131701 2854 leader_election.cc:302] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac4152d861cc4449b8c77a0dce86a16a; no voters:
I20260927 10:43:50.132256 2854 leader_election.cc:288] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260927 10:43:50.133025 2849 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4152d861cc4449b8c77a0dce86a16a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44901 } }
I20260927 10:43:50.133164 2849 leader_election.cc:302] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac4152d861cc4449b8c77a0dce86a16a; no voters:
I20260927 10:43:50.133196 2855 raft_consensus.cc:2741] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260927 10:43:50.133275 2849 leader_election.cc:288] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:43:50.133494 2854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:43:50.133586 2854 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a [term 1 LEADER]: Becoming Leader. State: Replica: ac4152d861cc4449b8c77a0dce86a16a, State: Running, Role: LEADER
I20260927 10:43:50.133915 2854 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4152d861cc4449b8c77a0dce86a16a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44901 } }
I20260927 10:43:50.134634 2849 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:43:50.135155 2856 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac4152d861cc4449b8c77a0dce86a16a. Latest consensus state: current_term: 1 leader_uuid: "ac4152d861cc4449b8c77a0dce86a16a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4152d861cc4449b8c77a0dce86a16a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44901 } } }
I20260927 10:43:50.135272 2856 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a [sys.catalog]: This master's current role is: LEADER
I20260927 10:43:50.135982 2855 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac4152d861cc4449b8c77a0dce86a16a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4152d861cc4449b8c77a0dce86a16a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44901 } } }
I20260927 10:43:50.136139 2855 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a [sys.catalog]: This master's current role is: LEADER
I20260927 10:43:50.136948 2859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:43:50.137645 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:50.122147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58784 (local address 127.0.98.62:44901)
0927 10:43:50.122612 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:50.122633 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:50.122674 (+ 41us) server_negotiation.cc:407] Connection header received
0927 10:43:50.123359 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:50.123377 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:50.123693 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:50.123981 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:50.124847 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:50.126054 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:50.127292 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:50.127798 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:50.127931 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:50.131198 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:50.131205 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:50.131232 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:50.131269 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:50.133361 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:50.133894 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:50.133900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:50.133906 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:50.134091 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:50.134439 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:50.134443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:50.134445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:50.134969 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:50.135178 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:50.136104 (+ 926us) server_negotiation.cc:299] Negotiation successful
0927 10:43:50.136467 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":195,"threads_started":1}
I20260927 10:43:50.139950 2859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:43:50.145150 2859 catalog_manager.cc:1347] Generated new cluster ID: 4fde5fd3476849a5a01121a457b3fd42
I20260927 10:43:50.145231 2859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:43:50.162048 2859 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:43:50.163084 2859 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:43:50.169160 2859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac4152d861cc4449b8c77a0dce86a16a: Generated new TSK 0
I20260927 10:43:50.169866 2859 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 10:43:50.201978 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44901
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:50.315408 2875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:50.315685 2875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:50.315728 2875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:50.320391 2875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:50.320482 2875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:50.320545 2875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:43:50.326258 2875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:50.327564 2875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:50.328645 2875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:50.339032 2883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:50.339044 2881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:50.339059 2880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:50.339210 2875 server_base.cc:1034] running on GCE node
I20260927 10:43:50.339823 2875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:50.340535 2875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:50.341722 2875 hybrid_clock.cc:648] HybridClock initialized: now 1790505830341707 us; error 40 us; skew 500 ppm
Sep 27 10:43:50 dist-test-slave-dzhw krb5kdc[2768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:50.344496 2875 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 10:43:50.345384 2875 webserver.cc:460] Webserver started at http://127.0.98.1:46245/ using document root <none> and password file <none>
I20260927 10:43:50.345912 2875 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:50.345968 2875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:50.346132 2875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:50.348428 2875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "31364121c9f44be8b6b28b7d023ca193"
format_stamp: "Formatted at 2026-09-27 10:43:50 on dist-test-slave-dzhw"
server_key: "f273abd3a27364c1adf599268bd71551"
server_key_iv: "d5c801da9e112ff2cfcc66238e1727fb"
server_key_version: "encryptionkey@0"
I20260927 10:43:50.348973 2875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "31364121c9f44be8b6b28b7d023ca193"
format_stamp: "Formatted at 2026-09-27 10:43:50 on dist-test-slave-dzhw"
server_key: "f273abd3a27364c1adf599268bd71551"
server_key_iv: "d5c801da9e112ff2cfcc66238e1727fb"
server_key_version: "encryptionkey@0"
I20260927 10:43:50.352797 2875 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:43:50.355687 2890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:50.356932 2875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:43:50.357059 2875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "31364121c9f44be8b6b28b7d023ca193"
format_stamp: "Formatted at 2026-09-27 10:43:50 on dist-test-slave-dzhw"
server_key: "f273abd3a27364c1adf599268bd71551"
server_key_iv: "d5c801da9e112ff2cfcc66238e1727fb"
server_key_version: "encryptionkey@0"
I20260927 10:43:50.357177 2875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:50.375633 2875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:50.379179 2875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:50.379411 2875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:50.380036 2875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:43:50.380877 2875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:50.380930 2875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:50.380970 2875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:50.381022 2875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:50.390403 2875 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38633
I20260927 10:43:50.390517 3003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38633 every 8 connection(s)
I20260927 10:43:50.391396 2875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:43:50.398708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 2875
I20260927 10:43:50.398811 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:43:50.399106 392 external_mini_cluster.cc:1428] Setting key d85981f988594eeb87dfb30ca1fd3f7b
Sep 27 10:43:50 dist-test-slave-dzhw krb5kdc[2768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:50.406389 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:50.393402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59603 (local address 127.0.98.62:44901)
0927 10:43:50.393568 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:50.393571 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:50.394496 (+ 925us) server_negotiation.cc:407] Connection header received
0927 10:43:50.395923 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:50.395927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:50.395988 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:50.396102 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:50.397335 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:50.397872 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:50.399270 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:50.399630 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:50.399764 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:50.402822 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:50.402825 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:50.402827 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:50.402851 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:50.404296 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:50.404841 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:50.404850 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:50.404851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:50.404929 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:50.405273 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:50.405276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:50.405278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:50.405463 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:50.405570 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:50.406078 (+ 508us) server_negotiation.cc:299] Negotiation successful
0927 10:43:50.406203 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 10:43:50.407307 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:50.393794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44901 (local address 127.0.98.1:59603)
0927 10:43:50.394306 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:50.394350 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:50.395695 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:50.396278 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:50.396285 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:50.396684 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:50.397182 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:50.397195 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:50.398017 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:50.398028 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:50.399121 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:50.399130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:50.399899 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:50.399905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:50.400103 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:50.400847 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:50.400870 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:50.402642 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:50.404440 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:50.404446 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:50.404463 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:50.404727 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:50.405022 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:50.405026 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:50.405028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:50.405172 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:50.405569 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:50.405576 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:50.405838 (+ 262us) client_negotiation.cc:769] Sending connection context
0927 10:43:50.406027 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 10:43:50.406363 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 10:43:50.408697 3004 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44901
I20260927 10:43:50.408950 3004 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:50.409456 3004 heartbeater.cc:511] Master 127.0.98.62:44901 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 10:43:50.411145 2811 ts_manager.cc:194] Registered new tserver with Master: 31364121c9f44be8b6b28b7d023ca193 (127.0.98.1:38633)
I20260927 10:43:50.412557 2811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 10:43:50.425836 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44901
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:50.537938 3009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:50.538203 3009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:50.538277 3009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:50.542321 3009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:50.542383 3009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:50.542461 3009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:43:50.547894 3009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:50.548941 3009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:50.549921 3009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:50.559713 3015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:50.559789 3009 server_base.cc:1034] running on GCE node
W20260927 10:43:50.559710 3014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:50.559717 3017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:50.560567 3009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:50.561363 3009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:50.562496 3009 hybrid_clock.cc:648] HybridClock initialized: now 1790505830562473 us; error 29 us; skew 500 ppm
Sep 27 10:43:50 dist-test-slave-dzhw krb5kdc[2768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:50.565364 3009 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 10:43:50.566318 3009 webserver.cc:460] Webserver started at http://127.0.98.2:40847/ using document root <none> and password file <none>
I20260927 10:43:50.566932 3009 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:50.567023 3009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:50.567276 3009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:50.569340 3009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "d06c4607eaf642609fe835b27a677877"
format_stamp: "Formatted at 2026-09-27 10:43:50 on dist-test-slave-dzhw"
server_key: "0b868879cbdc2c40e28f273eb78dd6fd"
server_key_iv: "b895a22abea28d360729425a5eee5706"
server_key_version: "encryptionkey@0"
I20260927 10:43:50.569914 3009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d06c4607eaf642609fe835b27a677877"
format_stamp: "Formatted at 2026-09-27 10:43:50 on dist-test-slave-dzhw"
server_key: "0b868879cbdc2c40e28f273eb78dd6fd"
server_key_iv: "b895a22abea28d360729425a5eee5706"
server_key_version: "encryptionkey@0"
I20260927 10:43:50.573757 3009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 10:43:50.576426 3024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:50.577669 3009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:43:50.577819 3009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "d06c4607eaf642609fe835b27a677877"
format_stamp: "Formatted at 2026-09-27 10:43:50 on dist-test-slave-dzhw"
server_key: "0b868879cbdc2c40e28f273eb78dd6fd"
server_key_iv: "b895a22abea28d360729425a5eee5706"
server_key_version: "encryptionkey@0"
I20260927 10:43:50.577942 3009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:50.592902 3009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:50.596451 3009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:50.596670 3009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:50.597301 3009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:43:50.598157 3009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:50.598232 3009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:50.598300 3009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:50.598343 3009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:50.607479 3009 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40351
I20260927 10:43:50.607502 3137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40351 every 8 connection(s)
I20260927 10:43:50.608402 3009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:43:50.612017 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 3009
I20260927 10:43:50.612144 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:43:50.612437 392 external_mini_cluster.cc:1428] Setting key 21aca253e1f6066ac8a50d149da7fcd7
Sep 27 10:43:50 dist-test-slave-dzhw krb5kdc[2768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 10:43:50.626206 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:50.610239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39155 (local address 127.0.98.62:44901)
0927 10:43:50.610386 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:50.610390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:50.611334 (+ 944us) server_negotiation.cc:407] Connection header received
0927 10:43:50.613017 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:50.613020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:50.613106 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:50.613257 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:50.615017 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:50.615828 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:50.617749 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:50.618318 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:50.618492 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:50.622341 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:50.622345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:50.622348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:50.622384 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:50.623943 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:50.624523 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:50.624526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:50.624528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:50.624614 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:50.625009 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:50.625012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:50.625014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:50.625210 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:50.625318 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:50.625917 (+ 599us) server_negotiation.cc:299] Negotiation successful
0927 10:43:50.626040 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 10:43:50.627293 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:50.610697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44901 (local address 127.0.98.2:39155)
0927 10:43:50.611163 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:50.611206 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:50.612671 (+ 1465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:50.613476 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:50.613485 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:50.614066 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:50.614851 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:50.614875 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:50.615999 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:50.616005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:50.617605 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:50.617619 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:50.618812 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:50.618819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:50.619157 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:50.620097 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:50.620124 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:50.622137 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:50.624091 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:50.624098 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:50.624119 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:50.624409 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:50.624713 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:50.624716 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:50.624718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:50.624906 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:50.625318 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:50.625326 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:50.625604 (+ 278us) client_negotiation.cc:769] Sending connection context
0927 10:43:50.625864 (+ 260us) client_negotiation.cc:240] Negotiation successful
0927 10:43:50.626285 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":121,"threads_started":1}
I20260927 10:43:50.628530 3138 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44901
I20260927 10:43:50.628827 3138 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:50.629441 3138 heartbeater.cc:511] Master 127.0.98.62:44901 requested a full tablet report, sending...
I20260927 10:43:50.630553 2811 ts_manager.cc:194] Registered new tserver with Master: d06c4607eaf642609fe835b27a677877 (127.0.98.2:40351)
I20260927 10:43:50.631135 2811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 10:43:50.641800 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44901
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:50.753517 3143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:50.753818 3143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:50.753861 3143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:50.758088 3143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:50.758150 3143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:50.758268 3143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:43:50.763724 3143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:50.764812 3143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:50.765870 3143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:50.775777 3149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:50.775849 3148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:50.775866 3143 server_base.cc:1034] running on GCE node
W20260927 10:43:50.775949 3151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:50.776422 3143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:50.777045 3143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:50.778251 3143 hybrid_clock.cc:648] HybridClock initialized: now 1790505830778233 us; error 29 us; skew 500 ppm
Sep 27 10:43:50 dist-test-slave-dzhw krb5kdc[2768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:50.781109 3143 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 10:43:50.782114 3143 webserver.cc:460] Webserver started at http://127.0.98.3:33769/ using document root <none> and password file <none>
I20260927 10:43:50.782749 3143 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:50.782850 3143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:50.783077 3143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:50.785107 3143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "e37c473627ee46bdbbffe751eb42e863"
format_stamp: "Formatted at 2026-09-27 10:43:50 on dist-test-slave-dzhw"
server_key: "3e397f6363dee53553c663aeb55838f2"
server_key_iv: "5d33ea57fd3b5d8452b650223e42cd34"
server_key_version: "encryptionkey@0"
I20260927 10:43:50.785672 3143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e37c473627ee46bdbbffe751eb42e863"
format_stamp: "Formatted at 2026-09-27 10:43:50 on dist-test-slave-dzhw"
server_key: "3e397f6363dee53553c663aeb55838f2"
server_key_iv: "5d33ea57fd3b5d8452b650223e42cd34"
server_key_version: "encryptionkey@0"
I20260927 10:43:50.789533 3143 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 10:43:50.792191 3158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:50.793545 3143 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:43:50.793704 3143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "e37c473627ee46bdbbffe751eb42e863"
format_stamp: "Formatted at 2026-09-27 10:43:50 on dist-test-slave-dzhw"
server_key: "3e397f6363dee53553c663aeb55838f2"
server_key_iv: "5d33ea57fd3b5d8452b650223e42cd34"
server_key_version: "encryptionkey@0"
I20260927 10:43:50.793838 3143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:50.815964 3143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:50.819067 3143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:50.819284 3143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:50.819897 3143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:43:50.820713 3143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:50.820794 3143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:50.820878 3143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:50.820927 3143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:50.829906 3143 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39781
I20260927 10:43:50.829926 3271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39781 every 8 connection(s)
I20260927 10:43:50.830892 3143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:43:50.838315 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 3143
I20260927 10:43:50.838478 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:43:50.838761 392 external_mini_cluster.cc:1428] Setting key 1413554949f4cf1f79ec49849f7212d8
Sep 27 10:43:50 dist-test-slave-dzhw krb5kdc[2768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:50.845280 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:50.832807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33069 (local address 127.0.98.62:44901)
0927 10:43:50.832962 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:50.832965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:50.833887 (+ 922us) server_negotiation.cc:407] Connection header received
0927 10:43:50.835486 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:50.835489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:50.835538 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:50.835615 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:50.836843 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:50.837375 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:50.838523 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:50.838790 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:50.838909 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:50.842011 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:50.842017 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:50.842022 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:50.842056 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:50.843507 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:50.843968 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:50.843971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:50.843972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:50.844040 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:50.844331 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:50.844333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:50.844334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:50.844497 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:50.844570 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:50.845056 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 10:43:50.845160 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 10:43:50.846093 3274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:50.833193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44901 (local address 127.0.98.3:33069)
0927 10:43:50.833735 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:50.833781 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:50.835236 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:50.835778 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:50.835784 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:50.836215 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:50.836707 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:50.836719 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:50.837483 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:50.837485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:50.838346 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:50.838354 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:50.838978 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:50.838983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:50.839146 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:50.839854 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:50.839870 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:50.841832 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:50.843609 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:50.843614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:50.843627 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:50.843880 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:50.844116 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:50.844118 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:50.844119 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:50.844253 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:50.844581 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:50.844586 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:50.844815 (+ 229us) client_negotiation.cc:769] Sending connection context
0927 10:43:50.844975 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 10:43:50.845224 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":113,"threads_started":1}
I20260927 10:43:50.847354 3272 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44901
I20260927 10:43:50.847607 3272 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:50.848106 3272 heartbeater.cc:511] Master 127.0.98.62:44901 requested a full tablet report, sending...
I20260927 10:43:50.849143 2811 ts_manager.cc:194] Registered new tserver with Master: e37c473627ee46bdbbffe751eb42e863 (127.0.98.3:39781)
I20260927 10:43:50.849917 2811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33069
I20260927 10:43:50.852918 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 10:43:50.866722 2852 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:43:50.865022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58800 (local address 127.0.98.62:44901)
0927 10:43:50.865178 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:50.865183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:50.865415 (+ 232us) server_negotiation.cc:407] Connection header received
0927 10:43:50.865517 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:50.865520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:50.865562 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:50.865633 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:50.866575 (+ 942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58800: BlockingRecv error: recv got EOF from 127.0.0.1:58800 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:43:50.867843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 2875
I20260927 10:43:50.877362 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 3009
I20260927 10:43:50.884696 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 3143
I20260927 10:43:50.891542 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 2781
2026-09-27T10:43:50Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2133 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:43:50 dist-test-slave-dzhw krb5kdc[3284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:43:50 dist-test-slave-dzhw krb5kdc[3284](info): set up 2 sockets
Sep 27 10:43:50 dist-test-slave-dzhw krb5kdc[3284](info): commencing operation
krb5kdc: starting...
W20260927 10:43:51.930320 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:43:52 dist-test-slave-dzhw krb5kdc[3284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:43:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:43:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:43:52.051040 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39575
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:52.159209 3297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:52.159483 3297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:52.159523 3297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:52.163581 3297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:43:52.163646 3297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:52.163666 3297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:52.163682 3297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:43:52.163697 3297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:43:52.168922 3297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:52.169981 3297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:52.171064 3297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:52.179350 3303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:52.179633 3297 server_base.cc:1034] running on GCE node
W20260927 10:43:52.179711 3305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:52.179359 3302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:52.180541 3297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:52.182065 3297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:52.183233 3297 hybrid_clock.cc:648] HybridClock initialized: now 1790505832183166 us; error 97 us; skew 500 ppm
Sep 27 10:43:52 dist-test-slave-dzhw krb5kdc[3284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:52.190670 3297 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:43:52.191624 3297 webserver.cc:460] Webserver started at http://127.0.98.62:43385/ using document root <none> and password file <none>
I20260927 10:43:52.192219 3297 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:52.192310 3297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:52.192530 3297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:52.194708 3297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "ab0110460d7248e5900285835b2b1338"
format_stamp: "Formatted at 2026-09-27 10:43:52 on dist-test-slave-dzhw"
server_key: "7c188c9925fe238f31aacb11942ab0e9"
server_key_iv: "1c3fcb5477fe6944be4f78838aea6f37"
server_key_version: "encryptionkey@0"
I20260927 10:43:52.195271 3297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "ab0110460d7248e5900285835b2b1338"
format_stamp: "Formatted at 2026-09-27 10:43:52 on dist-test-slave-dzhw"
server_key: "7c188c9925fe238f31aacb11942ab0e9"
server_key_iv: "1c3fcb5477fe6944be4f78838aea6f37"
server_key_version: "encryptionkey@0"
I20260927 10:43:52.198931 3297 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:43:52.201323 3312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:52.202611 3297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:43:52.202751 3297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "ab0110460d7248e5900285835b2b1338"
format_stamp: "Formatted at 2026-09-27 10:43:52 on dist-test-slave-dzhw"
server_key: "7c188c9925fe238f31aacb11942ab0e9"
server_key_iv: "1c3fcb5477fe6944be4f78838aea6f37"
server_key_version: "encryptionkey@0"
I20260927 10:43:52.202865 3297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:52.215948 3297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:52.218935 3297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:52.219136 3297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:52.226173 3297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39575
I20260927 10:43:52.226509 3364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39575 every 8 connection(s)
I20260927 10:43:52.227236 3297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:43:52.229926 3365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:52.235255 3365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab0110460d7248e5900285835b2b1338: Bootstrap starting.
I20260927 10:43:52.236840 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 3297
I20260927 10:43:52.236958 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:43:52.237221 392 external_mini_cluster.cc:1428] Setting key 5632a6b30fd409a51b80e13bbe009ac3
I20260927 10:43:52.237897 3365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab0110460d7248e5900285835b2b1338: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:52.238818 3365 log.cc:824] T 00000000000000000000000000000000 P ab0110460d7248e5900285835b2b1338: Log is configured to *not* fsync() on all Append() calls
I20260927 10:43:52.240592 3365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab0110460d7248e5900285835b2b1338: No bootstrap required, opened a new log
I20260927 10:43:52.244091 3365 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab0110460d7248e5900285835b2b1338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab0110460d7248e5900285835b2b1338" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } }
I20260927 10:43:52.244390 3365 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab0110460d7248e5900285835b2b1338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:52.244473 3365 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab0110460d7248e5900285835b2b1338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab0110460d7248e5900285835b2b1338, State: Initialized, Role: FOLLOWER
I20260927 10:43:52.244949 3365 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab0110460d7248e5900285835b2b1338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab0110460d7248e5900285835b2b1338" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } }
I20260927 10:43:52.245091 3365 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab0110460d7248e5900285835b2b1338 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:43:52.245162 3365 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab0110460d7248e5900285835b2b1338 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:43:52.245254 3365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab0110460d7248e5900285835b2b1338 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 10:43:52 dist-test-slave-dzhw krb5kdc[3284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:52.246210 3365 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab0110460d7248e5900285835b2b1338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab0110460d7248e5900285835b2b1338" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } }
I20260927 10:43:52.246593 3365 leader_election.cc:302] T 00000000000000000000000000000000 P ab0110460d7248e5900285835b2b1338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab0110460d7248e5900285835b2b1338; no voters:
I20260927 10:43:52.246981 3365 leader_election.cc:288] T 00000000000000000000000000000000 P ab0110460d7248e5900285835b2b1338 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:43:52.247105 3370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab0110460d7248e5900285835b2b1338 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:43:52.247351 3370 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab0110460d7248e5900285835b2b1338 [term 1 LEADER]: Becoming Leader. State: Replica: ab0110460d7248e5900285835b2b1338, State: Running, Role: LEADER
I20260927 10:43:52.247699 3370 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab0110460d7248e5900285835b2b1338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab0110460d7248e5900285835b2b1338" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } }
I20260927 10:43:52.248274 3365 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab0110460d7248e5900285835b2b1338 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:43:52.248986 3370 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab0110460d7248e5900285835b2b1338 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab0110460d7248e5900285835b2b1338. Latest consensus state: current_term: 1 leader_uuid: "ab0110460d7248e5900285835b2b1338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab0110460d7248e5900285835b2b1338" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } } }
I20260927 10:43:52.249109 3370 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab0110460d7248e5900285835b2b1338 [sys.catalog]: This master's current role is: LEADER
I20260927 10:43:52.249539 3371 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab0110460d7248e5900285835b2b1338 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab0110460d7248e5900285835b2b1338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab0110460d7248e5900285835b2b1338" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39575 } } }
I20260927 10:43:52.249691 3371 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab0110460d7248e5900285835b2b1338 [sys.catalog]: This master's current role is: LEADER
I20260927 10:43:52.249598 3379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:43:52.252254 3368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:52.238371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59268 (local address 127.0.98.62:39575)
0927 10:43:52.238843 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:52.238857 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:52.238892 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:43:52.239559 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:52.239575 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:52.239881 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:52.240219 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:52.241065 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:52.242028 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:52.242958 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:52.243454 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:52.243631 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:52.246254 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:52.246259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:52.246271 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:52.246298 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:52.248533 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:52.249101 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:52.249107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:52.249113 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:52.249305 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:52.249617 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:52.249621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:52.249623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:52.250187 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:52.250465 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:52.250747 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 10:43:52.251314 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":142,"threads_started":1}
I20260927 10:43:52.254350 3379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:43:52.255445 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:43:52.260584 3379 catalog_manager.cc:1347] Generated new cluster ID: ce3fa98d1a2d4b4fa91acc424c47a97c
I20260927 10:43:52.260649 3379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:43:52.271373 3379 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:43:52.272567 3368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:52.267174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59278 (local address 127.0.98.62:39575)
0927 10:43:52.267324 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:52.267329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:52.267528 (+ 199us) server_negotiation.cc:407] Connection header received
0927 10:43:52.267691 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:52.267694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:52.267745 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:52.267838 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:52.268356 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:52.269110 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:52.269931 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:52.270167 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:52.270279 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:52.271280 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:52.271283 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:52.271285 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:43:52.271884 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:52.271916 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:43:52.271934 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:52.272020 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:52.272188 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 10:43:52.272295 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":255,"server-negotiator.queue_time_us":67}
I20260927 10:43:52.272569 3379 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:43:52.288488 3379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab0110460d7248e5900285835b2b1338: Generated new TSK 0
I20260927 10:43:52.289250 3379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:43:52.295258 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 3297
2026-09-27T10:43:52Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1398 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:43:52 dist-test-slave-dzhw krb5kdc[3398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:43:52 dist-test-slave-dzhw krb5kdc[3398](info): set up 2 sockets
Sep 27 10:43:52 dist-test-slave-dzhw krb5kdc[3398](info): commencing operation
krb5kdc: starting...
W20260927 10:43:53.326502 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:43:53 dist-test-slave-dzhw krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:43:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:43:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:43:53.449621 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43219
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:53.559639 3411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:53.559954 3411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:53.560027 3411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:53.563987 3411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:43:53.564086 3411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:53.564126 3411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:53.564184 3411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:43:53.564225 3411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:43:53.569471 3411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:53.570600 3411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:53.571625 3411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:53.579998 3417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:53.580071 3416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:53.580179 3419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:53.580061 3411 server_base.cc:1034] running on GCE node
I20260927 10:43:53.580859 3411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:53.581976 3411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:53.583163 3411 hybrid_clock.cc:648] HybridClock initialized: now 1790505833583145 us; error 32 us; skew 500 ppm
Sep 27 10:43:53 dist-test-slave-dzhw krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:53.585907 3411 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:43:53.586937 3411 webserver.cc:460] Webserver started at http://127.0.98.62:33479/ using document root <none> and password file <none>
I20260927 10:43:53.587522 3411 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:53.587615 3411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:53.587829 3411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:53.589902 3411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "78d92c580ddf42299cb896abfaf823fe"
format_stamp: "Formatted at 2026-09-27 10:43:53 on dist-test-slave-dzhw"
server_key: "55e066c5c8e2b045d0477b7b9b9462ca"
server_key_iv: "e96edc32742be5107b325ee1d275385b"
server_key_version: "encryptionkey@0"
I20260927 10:43:53.590535 3411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "78d92c580ddf42299cb896abfaf823fe"
format_stamp: "Formatted at 2026-09-27 10:43:53 on dist-test-slave-dzhw"
server_key: "55e066c5c8e2b045d0477b7b9b9462ca"
server_key_iv: "e96edc32742be5107b325ee1d275385b"
server_key_version: "encryptionkey@0"
I20260927 10:43:53.594193 3411 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:43:53.596787 3426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:53.598024 3411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:43:53.598166 3411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "78d92c580ddf42299cb896abfaf823fe"
format_stamp: "Formatted at 2026-09-27 10:43:53 on dist-test-slave-dzhw"
server_key: "55e066c5c8e2b045d0477b7b9b9462ca"
server_key_iv: "e96edc32742be5107b325ee1d275385b"
server_key_version: "encryptionkey@0"
I20260927 10:43:53.598282 3411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:53.611507 3411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:53.614799 3411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:53.615041 3411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:53.622062 3411 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43219
I20260927 10:43:53.622354 3478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43219 every 8 connection(s)
I20260927 10:43:53.623162 3411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:43:53.625823 3479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:53.625850 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 3411
I20260927 10:43:53.625967 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:43:53.626259 392 external_mini_cluster.cc:1428] Setting key 7fca4cefe2c89a6ffa6d5151b1be48e0
I20260927 10:43:53.632375 3479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d92c580ddf42299cb896abfaf823fe: Bootstrap starting.
I20260927 10:43:53.635375 3479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78d92c580ddf42299cb896abfaf823fe: Neither blocks nor log segments found. Creating new log.
Sep 27 10:43:53 dist-test-slave-dzhw krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:53.636317 3479 log.cc:824] T 00000000000000000000000000000000 P 78d92c580ddf42299cb896abfaf823fe: Log is configured to *not* fsync() on all Append() calls
I20260927 10:43:53.638783 3479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d92c580ddf42299cb896abfaf823fe: No bootstrap required, opened a new log
I20260927 10:43:53.640955 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:53.627756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55432 (local address 127.0.98.62:43219)
0927 10:43:53.628201 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:53.628215 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:53.628252 (+ 37us) server_negotiation.cc:407] Connection header received
0927 10:43:53.629006 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:53.629024 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:53.629347 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:53.629727 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:53.630588 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:53.631561 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:53.632436 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:53.632992 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:53.633158 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:53.636242 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:53.636248 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:53.636263 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:53.636292 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:53.638160 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:53.638682 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:53.638687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:53.638692 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:53.638799 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:53.639097 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:53.639099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:53.639100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:53.639485 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:53.639628 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:53.639918 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 10:43:53.640213 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":131,"threads_started":1}
I20260927 10:43:53.642853 3479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78d92c580ddf42299cb896abfaf823fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d92c580ddf42299cb896abfaf823fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43219 } }
I20260927 10:43:53.643151 3479 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78d92c580ddf42299cb896abfaf823fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:53.643211 3479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78d92c580ddf42299cb896abfaf823fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d92c580ddf42299cb896abfaf823fe, State: Initialized, Role: FOLLOWER
I20260927 10:43:53.643648 3479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78d92c580ddf42299cb896abfaf823fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d92c580ddf42299cb896abfaf823fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43219 } }
I20260927 10:43:53.643801 3479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78d92c580ddf42299cb896abfaf823fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:43:53.643887 3479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78d92c580ddf42299cb896abfaf823fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:43:53.644004 3479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78d92c580ddf42299cb896abfaf823fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:53.644940 3479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78d92c580ddf42299cb896abfaf823fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d92c580ddf42299cb896abfaf823fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43219 } }
I20260927 10:43:53.645279 3479 leader_election.cc:302] T 00000000000000000000000000000000 P 78d92c580ddf42299cb896abfaf823fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78d92c580ddf42299cb896abfaf823fe; no voters:
I20260927 10:43:53.645617 3479 leader_election.cc:288] T 00000000000000000000000000000000 P 78d92c580ddf42299cb896abfaf823fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:43:53.645702 3484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78d92c580ddf42299cb896abfaf823fe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:43:53.645944 3484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78d92c580ddf42299cb896abfaf823fe [term 1 LEADER]: Becoming Leader. State: Replica: 78d92c580ddf42299cb896abfaf823fe, State: Running, Role: LEADER
I20260927 10:43:53.646234 3484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78d92c580ddf42299cb896abfaf823fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d92c580ddf42299cb896abfaf823fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43219 } }
I20260927 10:43:53.646919 3479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78d92c580ddf42299cb896abfaf823fe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:43:53.647851 3485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d92c580ddf42299cb896abfaf823fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78d92c580ddf42299cb896abfaf823fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d92c580ddf42299cb896abfaf823fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43219 } } }
I20260927 10:43:53.647980 3485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d92c580ddf42299cb896abfaf823fe [sys.catalog]: This master's current role is: LEADER
I20260927 10:43:53.648344 3486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d92c580ddf42299cb896abfaf823fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d92c580ddf42299cb896abfaf823fe. Latest consensus state: current_term: 1 leader_uuid: "78d92c580ddf42299cb896abfaf823fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d92c580ddf42299cb896abfaf823fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43219 } } }
I20260927 10:43:53.648439 3486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d92c580ddf42299cb896abfaf823fe [sys.catalog]: This master's current role is: LEADER
I20260927 10:43:53.648399 3493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:43:53.651268 3493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:43:53.656853 3493 catalog_manager.cc:1347] Generated new cluster ID: 5c039f21ff634262a9db6e058af8c882
I20260927 10:43:53.656935 3493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:43:53.670464 3493 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:43:53.671746 3493 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:43:53.676482 3493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78d92c580ddf42299cb896abfaf823fe: Generated new TSK 0
I20260927 10:43:53.677284 3493 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 10:43:53.715384 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43219
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:53.826906 3505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:53.827170 3505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:53.827234 3505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:53.831346 3505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:53.831408 3505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:53.831491 3505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:43:53.836786 3505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:53.837872 3505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:53.838971 3505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:53.848690 3513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:53.848708 3511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:53.848697 3510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:53.849116 3505 server_base.cc:1034] running on GCE node
I20260927 10:43:53.849633 3505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:53.850359 3505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:53.851572 3505 hybrid_clock.cc:648] HybridClock initialized: now 1790505833851549 us; error 47 us; skew 500 ppm
Sep 27 10:43:53 dist-test-slave-dzhw krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:53.854401 3505 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 10:43:53.855417 3505 webserver.cc:460] Webserver started at http://127.0.98.1:39685/ using document root <none> and password file <none>
I20260927 10:43:53.855985 3505 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:53.856076 3505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:53.856305 3505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:53.858345 3505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "568a961f9e714cd0942e0333c943c525"
format_stamp: "Formatted at 2026-09-27 10:43:53 on dist-test-slave-dzhw"
server_key: "7147a028e5a35e9b29a7c73a2ecb3003"
server_key_iv: "aafb55bd34241c1642feea56d7e9bad2"
server_key_version: "encryptionkey@0"
I20260927 10:43:53.858956 3505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "568a961f9e714cd0942e0333c943c525"
format_stamp: "Formatted at 2026-09-27 10:43:53 on dist-test-slave-dzhw"
server_key: "7147a028e5a35e9b29a7c73a2ecb3003"
server_key_iv: "aafb55bd34241c1642feea56d7e9bad2"
server_key_version: "encryptionkey@0"
I20260927 10:43:53.862588 3505 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:43:53.865314 3520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:53.866634 3505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:43:53.866765 3505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "568a961f9e714cd0942e0333c943c525"
format_stamp: "Formatted at 2026-09-27 10:43:53 on dist-test-slave-dzhw"
server_key: "7147a028e5a35e9b29a7c73a2ecb3003"
server_key_iv: "aafb55bd34241c1642feea56d7e9bad2"
server_key_version: "encryptionkey@0"
I20260927 10:43:53.866901 3505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:53.888537 3505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:53.891749 3505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:53.891916 3505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:53.892474 3505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:43:53.893249 3505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:53.893293 3505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:53.893332 3505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:53.893348 3505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:53.902279 3505 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39983
I20260927 10:43:53.902302 3633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39983 every 8 connection(s)
I20260927 10:43:53.903288 3505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:43:53.911888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 3505
I20260927 10:43:53.912045 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:43:53.912348 392 external_mini_cluster.cc:1428] Setting key 5b6d8a02cf8974b1038ded1004e11a29
Sep 27 10:43:53 dist-test-slave-dzhw krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:53.918495 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:53.905301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58723 (local address 127.0.98.62:43219)
0927 10:43:53.905456 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:53.905460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:53.906366 (+ 906us) server_negotiation.cc:407] Connection header received
0927 10:43:53.907911 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:53.907915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:53.907974 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:53.908089 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:53.909347 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:53.909891 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:53.911159 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:53.911397 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:53.911515 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:53.914184 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:53.914190 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:53.914196 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:53.914227 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:53.915948 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:53.916648 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:53.916651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:53.916652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:53.916750 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:53.917163 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:53.917169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:53.917173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:53.917437 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:53.917548 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:53.918076 (+ 528us) server_negotiation.cc:299] Negotiation successful
0927 10:43:53.918237 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 10:43:53.919400 3636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:53.905718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43219 (local address 127.0.98.1:58723)
0927 10:43:53.906205 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:53.906251 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:53.907664 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:53.908278 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:53.908286 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:53.908684 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:53.909187 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:53.909200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:53.910028 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:53.910031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:53.911027 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:53.911037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:53.911563 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:53.911567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:53.911743 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:53.912482 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:53.912511 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:53.914001 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:53.916074 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:53.916081 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:53.916105 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:53.916531 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:53.916848 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:53.916853 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:53.916857 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:53.917052 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:53.917548 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:53.917554 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:53.917810 (+ 256us) client_negotiation.cc:769] Sending connection context
0927 10:43:53.917990 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 10:43:53.918324 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":123,"threads_started":1}
I20260927 10:43:53.920799 3634 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43219
I20260927 10:43:53.921054 3634 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:53.921571 3634 heartbeater.cc:511] Master 127.0.98.62:43219 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 10:43:53.923280 3443 ts_manager.cc:194] Registered new tserver with Master: 568a961f9e714cd0942e0333c943c525 (127.0.98.1:39983)
I20260927 10:43:53.924450 3443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 10:43:53.941859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43219
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:54.053681 3639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:54.053964 3639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:54.054040 3639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:54.058386 3639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:54.058470 3639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:54.058531 3639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:43:54.063867 3639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:54.064956 3639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:54.065987 3639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:54.076283 3644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:54.076334 3639 server_base.cc:1034] running on GCE node
W20260927 10:43:54.076272 3645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:54.076318 3647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:54.076927 3639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:54.077621 3639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:54.078804 3639 hybrid_clock.cc:648] HybridClock initialized: now 1790505834078784 us; error 33 us; skew 500 ppm
Sep 27 10:43:54 dist-test-slave-dzhw krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:54.081501 3639 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 10:43:54.082378 3639 webserver.cc:460] Webserver started at http://127.0.98.2:38245/ using document root <none> and password file <none>
I20260927 10:43:54.082973 3639 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:54.083065 3639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:54.083295 3639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:54.085335 3639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "8e32a90241a4427f870f8f2a92de4957"
format_stamp: "Formatted at 2026-09-27 10:43:54 on dist-test-slave-dzhw"
server_key: "7149101e11e99a0b78dc560a4998047c"
server_key_iv: "c8759dc8994bb39ffef61b7069035975"
server_key_version: "encryptionkey@0"
I20260927 10:43:54.085902 3639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8e32a90241a4427f870f8f2a92de4957"
format_stamp: "Formatted at 2026-09-27 10:43:54 on dist-test-slave-dzhw"
server_key: "7149101e11e99a0b78dc560a4998047c"
server_key_iv: "c8759dc8994bb39ffef61b7069035975"
server_key_version: "encryptionkey@0"
I20260927 10:43:54.089638 3639 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 10:43:54.092262 3654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:54.093684 3639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:43:54.093835 3639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "8e32a90241a4427f870f8f2a92de4957"
format_stamp: "Formatted at 2026-09-27 10:43:54 on dist-test-slave-dzhw"
server_key: "7149101e11e99a0b78dc560a4998047c"
server_key_iv: "c8759dc8994bb39ffef61b7069035975"
server_key_version: "encryptionkey@0"
I20260927 10:43:54.093954 3639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:54.121210 3639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:54.124595 3639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:54.124814 3639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:54.125437 3639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:43:54.126255 3639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:54.126328 3639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:54.126418 3639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:54.126495 3639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:54.135452 3639 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35831
I20260927 10:43:54.135488 3767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35831 every 8 connection(s)
I20260927 10:43:54.136363 3639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:43:54.138685 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 3639
I20260927 10:43:54.138813 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:43:54.139147 392 external_mini_cluster.cc:1428] Setting key 5b633a343bc3b02152f67c2063b22e56
Sep 27 10:43:54 dist-test-slave-dzhw krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 10:43:54.151391 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:54.138244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39773 (local address 127.0.98.62:43219)
0927 10:43:54.138474 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:54.138480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:54.139568 (+ 1088us) server_negotiation.cc:407] Connection header received
0927 10:43:54.141288 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:54.141291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:54.141342 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:54.141427 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:54.142706 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:54.143224 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:54.144454 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:54.144674 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:54.144797 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:54.147433 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:54.147437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:54.147439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:54.147467 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:54.149223 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:54.149755 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:54.149759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:54.149760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:54.149846 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:54.150196 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:54.150199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:54.150201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:54.150397 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:54.150539 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:54.151160 (+ 621us) server_negotiation.cc:299] Negotiation successful
0927 10:43:54.151264 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 10:43:54.152649 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:54.138813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43219 (local address 127.0.98.2:39773)
0927 10:43:54.139407 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:54.139452 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:54.141069 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:54.141615 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:54.141622 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:54.141986 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:54.142556 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:54.142572 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:54.143355 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:54.143358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:54.144335 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:54.144344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:54.144836 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:54.144840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:54.145016 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:54.145645 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:54.145661 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:54.147269 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:54.149347 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:54.149353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:54.149370 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:54.149633 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:54.149940 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:54.149945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:54.149947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:54.150092 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:54.150549 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:54.150557 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:54.150866 (+ 309us) client_negotiation.cc:769] Sending connection context
0927 10:43:54.151075 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 10:43:54.151477 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":154,"threads_started":1}
I20260927 10:43:54.153877 3768 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43219
I20260927 10:43:54.154145 3768 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:54.154721 3768 heartbeater.cc:511] Master 127.0.98.62:43219 requested a full tablet report, sending...
I20260927 10:43:54.155831 3443 ts_manager.cc:194] Registered new tserver with Master: 8e32a90241a4427f870f8f2a92de4957 (127.0.98.2:35831)
I20260927 10:43:54.156741 3443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 10:43:54.167466 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43219
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:54.287266 3773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:54.287544 3773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:54.287587 3773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:54.291597 3773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:54.291663 3773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:54.291726 3773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:43:54.297050 3773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:54.298143 3773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:54.299149 3773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:54.308826 3779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:54.308972 3778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:54.308881 3781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:54.308854 3773 server_base.cc:1034] running on GCE node
I20260927 10:43:54.309439 3773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:54.310050 3773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:54.311218 3773 hybrid_clock.cc:648] HybridClock initialized: now 1790505834311188 us; error 42 us; skew 500 ppm
Sep 27 10:43:54 dist-test-slave-dzhw krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:54.314039 3773 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 10:43:54.314951 3773 webserver.cc:460] Webserver started at http://127.0.98.3:45193/ using document root <none> and password file <none>
I20260927 10:43:54.315503 3773 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:54.315589 3773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:54.315807 3773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:54.317802 3773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "40737acb5e5342f58cfdee1b4f4259f6"
format_stamp: "Formatted at 2026-09-27 10:43:54 on dist-test-slave-dzhw"
server_key: "e1e877d3cb65affade56710e9555f618"
server_key_iv: "9a4523315f533637f376dcb19ec97ae6"
server_key_version: "encryptionkey@0"
I20260927 10:43:54.318360 3773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "40737acb5e5342f58cfdee1b4f4259f6"
format_stamp: "Formatted at 2026-09-27 10:43:54 on dist-test-slave-dzhw"
server_key: "e1e877d3cb65affade56710e9555f618"
server_key_iv: "9a4523315f533637f376dcb19ec97ae6"
server_key_version: "encryptionkey@0"
I20260927 10:43:54.322114 3773 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:43:54.324906 3788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:54.326375 3773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 10:43:54.326589 3773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "40737acb5e5342f58cfdee1b4f4259f6"
format_stamp: "Formatted at 2026-09-27 10:43:54 on dist-test-slave-dzhw"
server_key: "e1e877d3cb65affade56710e9555f618"
server_key_iv: "9a4523315f533637f376dcb19ec97ae6"
server_key_version: "encryptionkey@0"
I20260927 10:43:54.326723 3773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:54.337627 3773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:54.341035 3773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:54.341239 3773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:54.341851 3773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:43:54.342742 3773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:54.342813 3773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:54.342885 3773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:54.342926 3773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:54.352058 3773 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33925
I20260927 10:43:54.352084 3901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33925 every 8 connection(s)
I20260927 10:43:54.353025 3773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:43:54.353813 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 3773
I20260927 10:43:54.353935 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:43:54.354195 392 external_mini_cluster.cc:1428] Setting key cbc25df9e14f85d0f47c5b24bf7fdc32
Sep 27 10:43:54 dist-test-slave-dzhw krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:54.366915 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:54.354963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50641 (local address 127.0.98.62:43219)
0927 10:43:54.355111 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:54.355115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:54.356188 (+ 1073us) server_negotiation.cc:407] Connection header received
0927 10:43:54.357623 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:54.357626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:54.357668 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:54.357738 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:54.358984 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:54.359509 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:54.360665 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:54.360885 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:54.360974 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:54.363503 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:54.363505 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:54.363508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:54.363530 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:54.364885 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:54.365399 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:54.365402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:54.365403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:54.365491 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:54.365835 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:54.365837 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:54.365838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:54.366010 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:54.366088 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:54.366635 (+ 547us) server_negotiation.cc:299] Negotiation successful
0927 10:43:54.366741 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:43:54.367857 3904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:54.355510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43219 (local address 127.0.98.3:50641)
0927 10:43:54.356031 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:54.356085 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:54.357434 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:54.357870 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:54.357876 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:54.358236 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:54.358855 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:54.358870 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:54.359622 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:54.359625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:54.360528 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:54.360537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:54.361018 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:54.361020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:54.361153 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:54.361792 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:54.361808 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:54.363347 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:54.364986 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:54.364991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:54.365004 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:54.365288 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:54.365605 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:54.365607 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:54.365609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:54.365747 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:54.366097 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:54.366101 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:54.366339 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 10:43:54.366572 (+ 233us) client_negotiation.cc:240] Negotiation successful
0927 10:43:54.366833 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":162,"threads_started":1}
I20260927 10:43:54.368989 3902 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43219
I20260927 10:43:54.369251 3902 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:54.369773 3902 heartbeater.cc:511] Master 127.0.98.62:43219 requested a full tablet report, sending...
I20260927 10:43:54.370844 3443 ts_manager.cc:194] Registered new tserver with Master: 40737acb5e5342f58cfdee1b4f4259f6 (127.0.98.3:33925)
I20260927 10:43:54.371471 3443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50641
I20260927 10:43:54.380170 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 10:43:54 dist-test-slave-dzhw krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505834, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 10:43:54 dist-test-slave-dzhw krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505834, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:54.405447 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:54.396949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60600 (local address 127.0.98.62:43219)
0927 10:43:54.397073 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:54.397076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:54.397238 (+ 162us) server_negotiation.cc:407] Connection header received
0927 10:43:54.397385 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:54.397388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:54.397431 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:54.397497 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:54.398221 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:54.398820 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:54.399616 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:54.399817 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:54.399892 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:54.402343 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:54.402346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:54.402348 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:54.402371 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:54.403891 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:54.404323 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:54.404326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:54.404329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:54.404422 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:54.404659 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:54.404661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:54.404663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:54.404870 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:54.404970 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:54.405148 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 10:43:54.405234 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 10:43:54.405927 3443 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60600
I20260927 10:43:54.406823 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 3505
I20260927 10:43:54.412689 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 3639
I20260927 10:43:54.418568 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 3773
I20260927 10:43:54.423842 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 3411
2026-09-27T10:43:54Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2130 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:43:54 dist-test-slave-dzhw krb5kdc[3914](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:43:54 dist-test-slave-dzhw krb5kdc[3914](info): set up 2 sockets
Sep 27 10:43:54 dist-test-slave-dzhw krb5kdc[3914](info): commencing operation
krb5kdc: starting...
W20260927 10:43:55.454507 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:43:55 dist-test-slave-dzhw krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:43:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:43:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:43:55.581334 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40891
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:55.693885 3927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:55.694213 3927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:55.694294 3927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:55.698292 3927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:43:55.698400 3927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:55.698457 3927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:55.698520 3927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:43:55.698556 3927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:43:55.704456 3927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:55.705552 3927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:55.706658 3927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:55.714994 3935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:55.714937 3933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:55.714957 3927 server_base.cc:1034] running on GCE node
W20260927 10:43:55.714937 3932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:55.715689 3927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:55.716786 3927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:55.717952 3927 hybrid_clock.cc:648] HybridClock initialized: now 1790505835717935 us; error 29 us; skew 500 ppm
Sep 27 10:43:55 dist-test-slave-dzhw krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:55.720872 3927 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:43:55.721809 3927 webserver.cc:460] Webserver started at http://127.0.98.62:42031/ using document root <none> and password file <none>
I20260927 10:43:55.722383 3927 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:55.722523 3927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:55.722755 3927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:55.724772 3927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "6c3634e3c1844edda5b05680b1043825"
format_stamp: "Formatted at 2026-09-27 10:43:55 on dist-test-slave-dzhw"
server_key: "fcf3297e754748600b17e8e4265958f1"
server_key_iv: "40ceea6aa7d00e390972100c581f159a"
server_key_version: "encryptionkey@0"
I20260927 10:43:55.725335 3927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "6c3634e3c1844edda5b05680b1043825"
format_stamp: "Formatted at 2026-09-27 10:43:55 on dist-test-slave-dzhw"
server_key: "fcf3297e754748600b17e8e4265958f1"
server_key_iv: "40ceea6aa7d00e390972100c581f159a"
server_key_version: "encryptionkey@0"
I20260927 10:43:55.729050 3927 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 10:43:55.731532 3942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:55.732728 3927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:43:55.732879 3927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "6c3634e3c1844edda5b05680b1043825"
format_stamp: "Formatted at 2026-09-27 10:43:55 on dist-test-slave-dzhw"
server_key: "fcf3297e754748600b17e8e4265958f1"
server_key_iv: "40ceea6aa7d00e390972100c581f159a"
server_key_version: "encryptionkey@0"
I20260927 10:43:55.732996 3927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:55.758291 3927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:55.761770 3927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:55.761991 3927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:55.769308 3927 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40891
I20260927 10:43:55.769327 3994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40891 every 8 connection(s)
I20260927 10:43:55.770287 3927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:43:55.773053 3995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:55.777557 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 3927
I20260927 10:43:55.777688 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:43:55.777959 392 external_mini_cluster.cc:1428] Setting key d6d903545f6d624a213dc2ce0c7372db
I20260927 10:43:55.778803 3995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c3634e3c1844edda5b05680b1043825: Bootstrap starting.
I20260927 10:43:55.781095 3995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c3634e3c1844edda5b05680b1043825: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:55.782054 3995 log.cc:824] T 00000000000000000000000000000000 P 6c3634e3c1844edda5b05680b1043825: Log is configured to *not* fsync() on all Append() calls
I20260927 10:43:55.784173 3995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c3634e3c1844edda5b05680b1043825: No bootstrap required, opened a new log
Sep 27 10:43:55 dist-test-slave-dzhw krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:55.788931 3995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c3634e3c1844edda5b05680b1043825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3634e3c1844edda5b05680b1043825" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40891 } }
I20260927 10:43:55.789239 3995 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c3634e3c1844edda5b05680b1043825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:55.789305 3995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c3634e3c1844edda5b05680b1043825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c3634e3c1844edda5b05680b1043825, State: Initialized, Role: FOLLOWER
I20260927 10:43:55.789893 3995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c3634e3c1844edda5b05680b1043825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3634e3c1844edda5b05680b1043825" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40891 } }
I20260927 10:43:55.790084 3995 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c3634e3c1844edda5b05680b1043825 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:43:55.790179 3995 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c3634e3c1844edda5b05680b1043825 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:43:55.790279 3995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c3634e3c1844edda5b05680b1043825 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:55.791461 3995 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c3634e3c1844edda5b05680b1043825 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3634e3c1844edda5b05680b1043825" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40891 } }
I20260927 10:43:55.791857 3995 leader_election.cc:302] T 00000000000000000000000000000000 P 6c3634e3c1844edda5b05680b1043825 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c3634e3c1844edda5b05680b1043825; no voters:
I20260927 10:43:55.792217 3995 leader_election.cc:288] T 00000000000000000000000000000000 P 6c3634e3c1844edda5b05680b1043825 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:43:55.792343 4000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c3634e3c1844edda5b05680b1043825 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:43:55.792599 4000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c3634e3c1844edda5b05680b1043825 [term 1 LEADER]: Becoming Leader. State: Replica: 6c3634e3c1844edda5b05680b1043825, State: Running, Role: LEADER
I20260927 10:43:55.792987 4000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c3634e3c1844edda5b05680b1043825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3634e3c1844edda5b05680b1043825" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40891 } }
I20260927 10:43:55.793327 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:55.779472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58408 (local address 127.0.98.62:40891)
0927 10:43:55.779916 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:55.779931 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:55.779965 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:43:55.780578 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:55.780594 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:55.780932 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:55.781252 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:55.782116 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:55.783355 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:55.784453 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:55.784920 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:55.785042 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:55.787847 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:55.787854 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:55.787872 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:55.787906 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:55.789842 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:55.790352 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:55.790358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:55.790365 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:55.790545 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:55.790858 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:55.790861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:55.790863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:55.791413 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:55.791611 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:55.791955 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 10:43:55.792275 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":144,"threads_started":1}
I20260927 10:43:55.793532 3995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c3634e3c1844edda5b05680b1043825 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:43:55.794698 4001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c3634e3c1844edda5b05680b1043825 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c3634e3c1844edda5b05680b1043825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3634e3c1844edda5b05680b1043825" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40891 } } }
I20260927 10:43:55.794817 4001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c3634e3c1844edda5b05680b1043825 [sys.catalog]: This master's current role is: LEADER
I20260927 10:43:55.795092 4002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c3634e3c1844edda5b05680b1043825 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c3634e3c1844edda5b05680b1043825. Latest consensus state: current_term: 1 leader_uuid: "6c3634e3c1844edda5b05680b1043825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3634e3c1844edda5b05680b1043825" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40891 } } }
I20260927 10:43:55.795172 4002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c3634e3c1844edda5b05680b1043825 [sys.catalog]: This master's current role is: LEADER
I20260927 10:43:55.795919 4009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:43:55.799448 4009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:43:55.805533 4009 catalog_manager.cc:1347] Generated new cluster ID: c3ebdee54ce6426f834ee560eebe912b
I20260927 10:43:55.805629 4009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:43:55.823778 4009 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:43:55.824800 4009 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:43:55.831774 4009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c3634e3c1844edda5b05680b1043825: Generated new TSK 0
I20260927 10:43:55.832494 4009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 10:43:55.867575 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40891
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:55.977376 4021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:55.977694 4021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:55.977772 4021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:55.981876 4021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:55.981980 4021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:55.982074 4021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:43:55.987461 4021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:55.988570 4021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:55.989609 4021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:55.999725 4026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:55.999729 4027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:55.999729 4029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:55.999799 4021 server_base.cc:1034] running on GCE node
I20260927 10:43:56.000373 4021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:56.001044 4021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:56.002252 4021 hybrid_clock.cc:648] HybridClock initialized: now 1790505836002238 us; error 33 us; skew 500 ppm
Sep 27 10:43:56 dist-test-slave-dzhw krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:56.005074 4021 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 10:43:56.005898 4021 webserver.cc:460] Webserver started at http://127.0.98.1:46201/ using document root <none> and password file <none>
I20260927 10:43:56.006460 4021 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:56.006532 4021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:56.006709 4021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:56.008715 4021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "d5a90a4f75f94df59f73cac1f9a37212"
format_stamp: "Formatted at 2026-09-27 10:43:56 on dist-test-slave-dzhw"
server_key: "a1755c9522a5c21878af1c8be1f2e935"
server_key_iv: "36194d4810d3fa51b4d19ff70fdb5341"
server_key_version: "encryptionkey@0"
I20260927 10:43:56.009220 4021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d5a90a4f75f94df59f73cac1f9a37212"
format_stamp: "Formatted at 2026-09-27 10:43:56 on dist-test-slave-dzhw"
server_key: "a1755c9522a5c21878af1c8be1f2e935"
server_key_iv: "36194d4810d3fa51b4d19ff70fdb5341"
server_key_version: "encryptionkey@0"
I20260927 10:43:56.012935 4021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 10:43:56.015429 4036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:56.016747 4021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:43:56.016855 4021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "d5a90a4f75f94df59f73cac1f9a37212"
format_stamp: "Formatted at 2026-09-27 10:43:56 on dist-test-slave-dzhw"
server_key: "a1755c9522a5c21878af1c8be1f2e935"
server_key_iv: "36194d4810d3fa51b4d19ff70fdb5341"
server_key_version: "encryptionkey@0"
I20260927 10:43:56.016969 4021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:56.032859 4021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:56.035912 4021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:56.036111 4021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:56.036636 4021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:43:56.037487 4021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:56.037533 4021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:56.037571 4021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:56.037593 4021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:56.046751 4021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35397
I20260927 10:43:56.046810 4149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35397 every 8 connection(s)
I20260927 10:43:56.047675 4021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:43:56.054278 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 4021
I20260927 10:43:56.054379 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:43:56.054706 392 external_mini_cluster.cc:1428] Setting key 8b5f76bf088fe832528536a1cbd8c31f
Sep 27 10:43:56 dist-test-slave-dzhw krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:56.062331 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:56.049434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46237 (local address 127.0.98.62:40891)
0927 10:43:56.049566 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:56.049570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:56.050302 (+ 732us) server_negotiation.cc:407] Connection header received
0927 10:43:56.051736 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:56.051740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:56.051798 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:56.051905 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:56.053044 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:56.053573 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:56.054834 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:56.055152 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:56.055281 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:56.058163 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:56.058166 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:56.058168 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:56.058191 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:56.059911 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:56.060603 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:56.060609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:56.060612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:56.060703 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:56.061104 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:56.061110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:56.061115 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:56.061391 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:56.061519 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:56.061990 (+ 471us) server_negotiation.cc:299] Negotiation successful
0927 10:43:56.062133 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:43:56.063140 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:56.049746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40891 (local address 127.0.98.1:46237)
0927 10:43:56.050155 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:56.050196 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:56.051526 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:56.052072 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:56.052078 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:56.052427 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:56.052905 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:56.052916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:56.053696 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:56.053700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:56.054694 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:56.054703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:56.055315 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:56.055319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:56.055496 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:56.056141 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:56.056161 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:56.057973 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:56.060025 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:56.060030 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:56.060048 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:56.060413 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:56.060835 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:56.060838 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:56.060839 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:56.060997 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:56.061511 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:56.061516 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:56.061746 (+ 230us) client_negotiation.cc:769] Sending connection context
0927 10:43:56.061908 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 10:43:56.062165 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 10:43:56.064390 4150 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40891
I20260927 10:43:56.064620 4150 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:56.065097 4150 heartbeater.cc:511] Master 127.0.98.62:40891 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 10:43:56.066814 3956 ts_manager.cc:194] Registered new tserver with Master: d5a90a4f75f94df59f73cac1f9a37212 (127.0.98.1:35397)
I20260927 10:43:56.067934 3956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 10:43:56.081570 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40891
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:56.196206 4155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:56.196468 4155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:56.196537 4155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:56.200652 4155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:56.200711 4155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:56.200811 4155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:43:56.206089 4155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:56.207223 4155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:56.208238 4155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:56.217643 4160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:56.217643 4163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:56.217700 4155 server_base.cc:1034] running on GCE node
W20260927 10:43:56.217644 4161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:56.218289 4155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:56.218968 4155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:56.220175 4155 hybrid_clock.cc:648] HybridClock initialized: now 1790505836220158 us; error 38 us; skew 500 ppm
Sep 27 10:43:56 dist-test-slave-dzhw krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:56.223057 4155 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 10:43:56.223913 4155 webserver.cc:460] Webserver started at http://127.0.98.2:32983/ using document root <none> and password file <none>
I20260927 10:43:56.224479 4155 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:56.224567 4155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:56.224792 4155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:56.226846 4155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "407d5cfae7d44563ac86df334fc6f627"
format_stamp: "Formatted at 2026-09-27 10:43:56 on dist-test-slave-dzhw"
server_key: "8287a2db3879f525b3f894e6d45a65b6"
server_key_iv: "c95ce96443248d39cfdf5b26e437ca3f"
server_key_version: "encryptionkey@0"
I20260927 10:43:56.227427 4155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "407d5cfae7d44563ac86df334fc6f627"
format_stamp: "Formatted at 2026-09-27 10:43:56 on dist-test-slave-dzhw"
server_key: "8287a2db3879f525b3f894e6d45a65b6"
server_key_iv: "c95ce96443248d39cfdf5b26e437ca3f"
server_key_version: "encryptionkey@0"
I20260927 10:43:56.231118 4155 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:43:56.233575 4170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:56.234910 4155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:43:56.235038 4155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "407d5cfae7d44563ac86df334fc6f627"
format_stamp: "Formatted at 2026-09-27 10:43:56 on dist-test-slave-dzhw"
server_key: "8287a2db3879f525b3f894e6d45a65b6"
server_key_iv: "c95ce96443248d39cfdf5b26e437ca3f"
server_key_version: "encryptionkey@0"
I20260927 10:43:56.235159 4155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:56.256124 4155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:56.259178 4155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:56.259395 4155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:56.259984 4155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:43:56.260952 4155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:56.261013 4155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:56.261076 4155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:56.261123 4155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:56.269775 4155 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39767
I20260927 10:43:56.269806 4283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39767 every 8 connection(s)
I20260927 10:43:56.270714 4155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:43:56.278113 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 4155
I20260927 10:43:56.278254 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:43:56.278607 392 external_mini_cluster.cc:1428] Setting key a8ad88f11253df0f99d2beccfe704f9c
Sep 27 10:43:56 dist-test-slave-dzhw krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:56.284720 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:56.272475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49205 (local address 127.0.98.62:40891)
0927 10:43:56.272604 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:56.272608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:56.273404 (+ 796us) server_negotiation.cc:407] Connection header received
0927 10:43:56.274847 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:56.274850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:56.274908 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:56.275010 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:56.276128 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:56.276655 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:56.277789 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:56.278025 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:56.278134 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:56.280900 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:56.280907 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:56.280913 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:56.280946 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:56.282500 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:56.283111 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:56.283114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:56.283115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:56.283185 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:56.283567 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:56.283570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:56.283571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:56.283734 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:56.283816 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:56.284495 (+ 679us) server_negotiation.cc:299] Negotiation successful
0927 10:43:56.284603 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:43:56.286013 4286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:56.272811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40891 (local address 127.0.98.2:49205)
0927 10:43:56.273243 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:56.273302 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:56.274635 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:56.275176 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:56.275183 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:56.275537 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:56.275991 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:56.276002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:56.276780 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:56.276785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:56.277657 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:56.277665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:56.278173 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:56.278177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:56.278405 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:56.279077 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:56.279093 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:56.280735 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:56.282610 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:56.282618 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:56.282637 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:56.283002 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:56.283272 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:56.283277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:56.283283 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:56.283474 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:56.283850 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:56.283860 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:56.284216 (+ 356us) client_negotiation.cc:769] Sending connection context
0927 10:43:56.284455 (+ 239us) client_negotiation.cc:240] Negotiation successful
0927 10:43:56.284829 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":100,"threads_started":1}
I20260927 10:43:56.287199 4284 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40891
I20260927 10:43:56.287454 4284 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:56.287942 4284 heartbeater.cc:511] Master 127.0.98.62:40891 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 10:43:56.289031 3956 ts_manager.cc:194] Registered new tserver with Master: 407d5cfae7d44563ac86df334fc6f627 (127.0.98.2:39767)
I20260927 10:43:56.289706 3956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 10:43:56.305969 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40891
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:56.417594 4289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:56.417881 4289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:56.417949 4289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:56.422098 4289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:56.422178 4289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:56.422261 4289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:43:56.427551 4289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:56.428627 4289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:56.429633 4289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:56.439939 4294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:56.440034 4297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:56.440001 4289 server_base.cc:1034] running on GCE node
W20260927 10:43:56.439942 4295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:56.440624 4289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:56.441247 4289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:56.442401 4289 hybrid_clock.cc:648] HybridClock initialized: now 1790505836442386 us; error 31 us; skew 500 ppm
Sep 27 10:43:56 dist-test-slave-dzhw krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:56.445238 4289 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 10:43:56.446152 4289 webserver.cc:460] Webserver started at http://127.0.98.3:45161/ using document root <none> and password file <none>
I20260927 10:43:56.446754 4289 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:56.446818 4289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:56.446990 4289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:56.448968 4289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "abd76c5ad140415cb8ab1c2549eac860"
format_stamp: "Formatted at 2026-09-27 10:43:56 on dist-test-slave-dzhw"
server_key: "42fb3991b18d8b27f8103111b7a2b4d8"
server_key_iv: "6e9a757757348d86f380a49b4bc26f7f"
server_key_version: "encryptionkey@0"
I20260927 10:43:56.449473 4289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "abd76c5ad140415cb8ab1c2549eac860"
format_stamp: "Formatted at 2026-09-27 10:43:56 on dist-test-slave-dzhw"
server_key: "42fb3991b18d8b27f8103111b7a2b4d8"
server_key_iv: "6e9a757757348d86f380a49b4bc26f7f"
server_key_version: "encryptionkey@0"
I20260927 10:43:56.453076 4289 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:43:56.455495 4304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:56.456676 4289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:43:56.456794 4289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "abd76c5ad140415cb8ab1c2549eac860"
format_stamp: "Formatted at 2026-09-27 10:43:56 on dist-test-slave-dzhw"
server_key: "42fb3991b18d8b27f8103111b7a2b4d8"
server_key_iv: "6e9a757757348d86f380a49b4bc26f7f"
server_key_version: "encryptionkey@0"
I20260927 10:43:56.456902 4289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:56.483127 4289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:56.486392 4289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:56.486681 4289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:56.487318 4289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:43:56.488171 4289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:56.488241 4289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:56.488306 4289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:56.488530 4289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:56.498049 4289 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33379
I20260927 10:43:56.498070 4417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33379 every 8 connection(s)
I20260927 10:43:56.499006 4289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:43:56.502617 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 4289
I20260927 10:43:56.502770 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:43:56.503072 392 external_mini_cluster.cc:1428] Setting key 68d113bb9ba7a10dd23a1b3b9d889ef2
Sep 27 10:43:56 dist-test-slave-dzhw krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:56.514542 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:56.500933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37027 (local address 127.0.98.62:40891)
0927 10:43:56.501068 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:56.501072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:56.501831 (+ 759us) server_negotiation.cc:407] Connection header received
0927 10:43:56.503633 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:56.503636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:56.503702 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:56.503802 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:56.505294 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:56.505814 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:56.507003 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:56.507222 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:56.507306 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:56.510532 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:56.510538 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:56.510543 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:56.510573 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:56.512399 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:56.512945 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:56.512948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:56.512950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:56.513039 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:56.513389 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:56.513392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:56.513393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:56.513587 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:56.513693 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:56.514206 (+ 513us) server_negotiation.cc:299] Negotiation successful
0927 10:43:56.514338 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:43:56.515426 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:56.501256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40891 (local address 127.0.98.3:37027)
0927 10:43:56.501678 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:56.501721 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:56.503403 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:56.504018 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:56.504029 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:56.504531 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:56.505163 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:56.505179 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:56.505933 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:56.505935 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:56.506903 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:56.506912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:56.507398 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:56.507402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:56.507564 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:56.508138 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:56.508156 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:56.510267 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:56.512528 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:56.512534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:56.512551 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:56.512834 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:56.513130 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:56.513134 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:56.513136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:56.513283 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:56.513691 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:56.513698 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:56.513966 (+ 268us) client_negotiation.cc:769] Sending connection context
0927 10:43:56.514158 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 10:43:56.514512 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":105,"threads_started":1}
I20260927 10:43:56.516633 4418 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40891
I20260927 10:43:56.516851 4418 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:56.517289 4418 heartbeater.cc:511] Master 127.0.98.62:40891 requested a full tablet report, sending...
I20260927 10:43:56.518196 3956 ts_manager.cc:194] Registered new tserver with Master: abd76c5ad140415cb8ab1c2549eac860 (127.0.98.3:33379)
I20260927 10:43:56.518786 3956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37027
I20260927 10:43:56.530071 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 10:43:56 dist-test-slave-dzhw krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 10:43:56 dist-test-slave-dzhw krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 10:43:56.556428 4425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:56.546825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53276 (local address 127.0.98.1:35397)
0927 10:43:56.547129 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:56.547149 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:56.547198 (+ 49us) server_negotiation.cc:407] Connection header received
0927 10:43:56.547269 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:56.547276 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:56.547494 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:56.547662 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:56.548409 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:56.549284 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:56.550225 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:56.550681 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:56.550815 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:56.553271 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:56.553281 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:56.553294 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:56.553316 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:56.554891 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:56.555271 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:56.555276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:56.555279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:56.555355 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:56.555573 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:56.555575 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:56.555577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:56.555812 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:56.555980 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:56.556151 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 10:43:56.556273 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
Sep 27 10:43:56 dist-test-slave-dzhw krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:56.567775 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:56.559396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58424 (local address 127.0.98.62:40891)
0927 10:43:56.559563 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:56.559567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:56.559591 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:43:56.559656 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:56.559658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:56.559697 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:56.559762 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:56.560335 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:56.560973 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:56.561863 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:56.562069 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:56.562147 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:56.564512 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:56.564516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:56.564519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:56.564553 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:56.566063 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:56.566520 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:56.566524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:56.566526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:56.566611 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:56.566889 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:56.566894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:56.566896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:56.567118 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:56.567273 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:56.567496 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 10:43:56.567607 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 10:43:56.568179 3956 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58424
I20260927 10:43:56.568766 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 4021
I20260927 10:43:56.574996 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 4155
I20260927 10:43:56.580509 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 4289
I20260927 10:43:56.585741 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 3927
2026-09-27T10:43:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2161 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:43:56 dist-test-slave-dzhw krb5kdc[4430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:43:56 dist-test-slave-dzhw krb5kdc[4430](info): set up 2 sockets
Sep 27 10:43:56 dist-test-slave-dzhw krb5kdc[4430](info): commencing operation
krb5kdc: starting...
W20260927 10:43:57.650564 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:43:57 dist-test-slave-dzhw krb5kdc[4430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:43:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:43:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:43:57.773200 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35051
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:57.883505 4443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:57.883781 4443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:57.883831 4443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:57.887967 4443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:43:57.888037 4443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:57.888057 4443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:57.888074 4443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:43:57.888090 4443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:43:57.893508 4443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:57.894670 4443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:57.895710 4443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:57.904139 4448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:57.904105 4443 server_base.cc:1034] running on GCE node
W20260927 10:43:57.904090 4449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:57.904124 4451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:57.904984 4443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:57.906103 4443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:57.907281 4443 hybrid_clock.cc:648] HybridClock initialized: now 1790505837907282 us; error 53 us; skew 500 ppm
Sep 27 10:43:57 dist-test-slave-dzhw krb5kdc[4430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:57.915025 4443 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:43:57.915840 4443 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:57.915941 4443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:57.916185 4443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:57.918310 4443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "b041a4132352423f87e77240e3e0722b"
format_stamp: "Formatted at 2026-09-27 10:43:57 on dist-test-slave-dzhw"
server_key: "2736c068961c14d2f8b31a4ddfc8e973"
server_key_iv: "fcf284636fb2a279921c3260f4133852"
server_key_version: "encryptionkey@0"
I20260927 10:43:57.918926 4443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "b041a4132352423f87e77240e3e0722b"
format_stamp: "Formatted at 2026-09-27 10:43:57 on dist-test-slave-dzhw"
server_key: "2736c068961c14d2f8b31a4ddfc8e973"
server_key_iv: "fcf284636fb2a279921c3260f4133852"
server_key_version: "encryptionkey@0"
I20260927 10:43:57.922580 4443 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:43:57.925108 4457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:57.926306 4443 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:43:57.926479 4443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "b041a4132352423f87e77240e3e0722b"
format_stamp: "Formatted at 2026-09-27 10:43:57 on dist-test-slave-dzhw"
server_key: "2736c068961c14d2f8b31a4ddfc8e973"
server_key_iv: "fcf284636fb2a279921c3260f4133852"
server_key_version: "encryptionkey@0"
I20260927 10:43:57.926604 4443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:57.936194 4443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:57.939430 4443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:57.939638 4443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:57.946550 4443 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35051
I20260927 10:43:57.946822 4509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35051 every 8 connection(s)
I20260927 10:43:57.947260 4443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:43:57.949379 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 4443
I20260927 10:43:57.949522 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:43:57.949813 392 external_mini_cluster.cc:1428] Setting key 0d1cea42bc363ef8d2993067f5e2c359
I20260927 10:43:57.950059 4510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:57.956866 4510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b041a4132352423f87e77240e3e0722b: Bootstrap starting.
Sep 27 10:43:57 dist-test-slave-dzhw krb5kdc[4430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:57.959640 4510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b041a4132352423f87e77240e3e0722b: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:57.960342 4510 log.cc:824] T 00000000000000000000000000000000 P b041a4132352423f87e77240e3e0722b: Log is configured to *not* fsync() on all Append() calls
I20260927 10:43:57.962539 4510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b041a4132352423f87e77240e3e0722b: No bootstrap required, opened a new log
I20260927 10:43:57.965140 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:57.951191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43498 (local address 127.0.98.62:35051)
0927 10:43:57.951582 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:57.951599 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:57.951645 (+ 46us) server_negotiation.cc:407] Connection header received
0927 10:43:57.952699 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:57.952718 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:57.953073 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:57.953353 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:57.954259 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:57.955394 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:57.956566 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:57.957114 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:57.957278 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:57.960025 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:57.960030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:57.960044 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:57.960071 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:57.962211 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:57.962736 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:57.962742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:57.962748 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:57.962865 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:57.963154 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:57.963157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:57.963159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:57.963554 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:57.963754 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:57.964030 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 10:43:57.964377 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":261,"thread_start_us":133,"threads_started":1}
I20260927 10:43:57.966233 4510 raft_consensus.cc:348] T 00000000000000000000000000000000 P b041a4132352423f87e77240e3e0722b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b041a4132352423f87e77240e3e0722b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35051 } }
I20260927 10:43:57.966502 4510 raft_consensus.cc:374] T 00000000000000000000000000000000 P b041a4132352423f87e77240e3e0722b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:57.966567 4510 raft_consensus.cc:729] T 00000000000000000000000000000000 P b041a4132352423f87e77240e3e0722b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b041a4132352423f87e77240e3e0722b, State: Initialized, Role: FOLLOWER
I20260927 10:43:57.967020 4510 consensus_queue.cc:260] T 00000000000000000000000000000000 P b041a4132352423f87e77240e3e0722b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b041a4132352423f87e77240e3e0722b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35051 } }
I20260927 10:43:57.967199 4510 raft_consensus.cc:388] T 00000000000000000000000000000000 P b041a4132352423f87e77240e3e0722b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:43:57.967314 4510 raft_consensus.cc:482] T 00000000000000000000000000000000 P b041a4132352423f87e77240e3e0722b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:43:57.967417 4510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b041a4132352423f87e77240e3e0722b [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:57.968334 4510 raft_consensus.cc:504] T 00000000000000000000000000000000 P b041a4132352423f87e77240e3e0722b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b041a4132352423f87e77240e3e0722b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35051 } }
I20260927 10:43:57.968664 4510 leader_election.cc:302] T 00000000000000000000000000000000 P b041a4132352423f87e77240e3e0722b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b041a4132352423f87e77240e3e0722b; no voters:
I20260927 10:43:57.968998 4510 leader_election.cc:288] T 00000000000000000000000000000000 P b041a4132352423f87e77240e3e0722b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:43:57.969125 4515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b041a4132352423f87e77240e3e0722b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:43:57.969379 4515 raft_consensus.cc:686] T 00000000000000000000000000000000 P b041a4132352423f87e77240e3e0722b [term 1 LEADER]: Becoming Leader. State: Replica: b041a4132352423f87e77240e3e0722b, State: Running, Role: LEADER
I20260927 10:43:57.969712 4515 consensus_queue.cc:237] T 00000000000000000000000000000000 P b041a4132352423f87e77240e3e0722b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b041a4132352423f87e77240e3e0722b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35051 } }
I20260927 10:43:57.970178 4510 sys_catalog.cc:564] T 00000000000000000000000000000000 P b041a4132352423f87e77240e3e0722b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:43:57.971391 4516 sys_catalog.cc:455] T 00000000000000000000000000000000 P b041a4132352423f87e77240e3e0722b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b041a4132352423f87e77240e3e0722b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b041a4132352423f87e77240e3e0722b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35051 } } }
I20260927 10:43:57.971525 4516 sys_catalog.cc:458] T 00000000000000000000000000000000 P b041a4132352423f87e77240e3e0722b [sys.catalog]: This master's current role is: LEADER
I20260927 10:43:57.971830 4517 sys_catalog.cc:455] T 00000000000000000000000000000000 P b041a4132352423f87e77240e3e0722b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b041a4132352423f87e77240e3e0722b. Latest consensus state: current_term: 1 leader_uuid: "b041a4132352423f87e77240e3e0722b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b041a4132352423f87e77240e3e0722b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35051 } } }
I20260927 10:43:57.971902 4524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:43:57.971971 4517 sys_catalog.cc:458] T 00000000000000000000000000000000 P b041a4132352423f87e77240e3e0722b [sys.catalog]: This master's current role is: LEADER
I20260927 10:43:57.974742 4524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:43:57.980255 4524 catalog_manager.cc:1347] Generated new cluster ID: 5583d20129bf48eba4595bc23f53af8a
I20260927 10:43:57.980327 4524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:43:58.000120 4524 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:43:58.001263 4524 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:43:58.008742 4524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b041a4132352423f87e77240e3e0722b: Generated new TSK 0
I20260927 10:43:58.009536 4524 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 10:43:58.050642 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35051
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:58.163200 4536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:58.163492 4536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:58.163540 4536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:58.167863 4536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:58.167934 4536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:58.168028 4536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:43:58.173696 4536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:58.174820 4536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:58.175881 4536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:58.189357 4542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:58.189327 4541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:58.189504 4536 server_base.cc:1034] running on GCE node
W20260927 10:43:58.189338 4544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:58.190047 4536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:58.190852 4536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:58.192045 4536 hybrid_clock.cc:648] HybridClock initialized: now 1790505838192011 us; error 55 us; skew 500 ppm
Sep 27 10:43:58 dist-test-slave-dzhw krb5kdc[4430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:58.194902 4536 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 10:43:58.195612 4536 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:58.195676 4536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:58.195842 4536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:58.197983 4536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "81297263c3754576bdfbb5e9a0b7b522"
format_stamp: "Formatted at 2026-09-27 10:43:58 on dist-test-slave-dzhw"
server_key: "e34701af8ee2959ef6ebb4ad9b6a2636"
server_key_iv: "85d5fe95f06a5c44dad02d2529a7070a"
server_key_version: "encryptionkey@0"
I20260927 10:43:58.198592 4536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "81297263c3754576bdfbb5e9a0b7b522"
format_stamp: "Formatted at 2026-09-27 10:43:58 on dist-test-slave-dzhw"
server_key: "e34701af8ee2959ef6ebb4ad9b6a2636"
server_key_iv: "85d5fe95f06a5c44dad02d2529a7070a"
server_key_version: "encryptionkey@0"
I20260927 10:43:58.202287 4536 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:43:58.204826 4550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:58.206130 4536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:43:58.206243 4536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "81297263c3754576bdfbb5e9a0b7b522"
format_stamp: "Formatted at 2026-09-27 10:43:58 on dist-test-slave-dzhw"
server_key: "e34701af8ee2959ef6ebb4ad9b6a2636"
server_key_iv: "85d5fe95f06a5c44dad02d2529a7070a"
server_key_version: "encryptionkey@0"
I20260927 10:43:58.206404 4536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:58.236225 4536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:58.239723 4536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:58.239938 4536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:58.240554 4536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:43:58.241391 4536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:58.241437 4536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:58.241504 4536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:58.241534 4536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:58.250279 4536 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34879
I20260927 10:43:58.250298 4663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34879 every 8 connection(s)
I20260927 10:43:58.250959 4536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:43:58.257395 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 4536
I20260927 10:43:58.257504 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:43:58.257812 392 external_mini_cluster.cc:1428] Setting key c96d2b85a4c8bfb4dcc19e87b1400c1c
Sep 27 10:43:58 dist-test-slave-dzhw krb5kdc[4430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:58.266888 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:58.252841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34263 (local address 127.0.98.62:35051)
0927 10:43:58.253015 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:58.253019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:58.253854 (+ 835us) server_negotiation.cc:407] Connection header received
0927 10:43:58.255305 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:58.255309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:58.255372 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:58.255477 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:58.256696 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.257273 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.258778 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.259015 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.259132 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:58.262170 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:58.262176 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:58.262180 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:58.262225 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:58.264206 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:58.264799 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:58.264802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:58.264803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:58.264892 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:58.265439 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:58.265445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:58.265450 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:58.265741 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:58.265879 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:58.266508 (+ 629us) server_negotiation.cc:299] Negotiation successful
0927 10:43:58.266664 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 10:43:58.267747 4666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:58.253209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35051 (local address 127.0.98.1:34263)
0927 10:43:58.253673 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:58.253728 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:58.255049 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:58.255663 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:58.255670 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:58.256057 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:58.256546 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:58.256559 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.257405 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.257413 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:58.258647 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:58.258657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.259175 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.259179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:58.259419 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:58.260143 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:58.260170 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:58.261961 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:58.264347 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:58.264353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:58.264370 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:58.264671 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:58.265069 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:58.265075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:58.265080 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:58.265327 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:58.265894 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:58.265900 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:58.266177 (+ 277us) client_negotiation.cc:769] Sending connection context
0927 10:43:58.266385 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 10:43:58.266765 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 10:43:58.269248 4664 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35051
I20260927 10:43:58.269533 4664 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:58.270076 4664 heartbeater.cc:511] Master 127.0.98.62:35051 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 10:43:58.271898 4473 ts_manager.cc:194] Registered new tserver with Master: 81297263c3754576bdfbb5e9a0b7b522 (127.0.98.1:34879)
I20260927 10:43:58.273030 4473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 10:43:58.287752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35051
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:58.399060 4669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:58.399348 4669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:58.399428 4669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:58.403558 4669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:58.403626 4669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:58.403721 4669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:43:58.409142 4669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:58.410249 4669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:58.411345 4669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:58.421167 4674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:58.421373 4675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:58.421409 4669 server_base.cc:1034] running on GCE node
W20260927 10:43:58.421167 4677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:58.421890 4669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:58.422657 4669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:58.423846 4669 hybrid_clock.cc:648] HybridClock initialized: now 1790505838423808 us; error 51 us; skew 500 ppm
Sep 27 10:43:58 dist-test-slave-dzhw krb5kdc[4430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:58.426759 4669 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 10:43:58.427574 4669 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:58.427671 4669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:58.427933 4669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:58.430054 4669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "ee39f7e908104bcb82b9a26cf24c5046"
format_stamp: "Formatted at 2026-09-27 10:43:58 on dist-test-slave-dzhw"
server_key: "8e5e433a24125ae01aac04cd90bb3afd"
server_key_iv: "16591c374fc58505ee76f7ac4b171e55"
server_key_version: "encryptionkey@0"
I20260927 10:43:58.430696 4669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ee39f7e908104bcb82b9a26cf24c5046"
format_stamp: "Formatted at 2026-09-27 10:43:58 on dist-test-slave-dzhw"
server_key: "8e5e433a24125ae01aac04cd90bb3afd"
server_key_iv: "16591c374fc58505ee76f7ac4b171e55"
server_key_version: "encryptionkey@0"
I20260927 10:43:58.434365 4669 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:43:58.436860 4683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:58.438113 4669 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:43:58.438248 4669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "ee39f7e908104bcb82b9a26cf24c5046"
format_stamp: "Formatted at 2026-09-27 10:43:58 on dist-test-slave-dzhw"
server_key: "8e5e433a24125ae01aac04cd90bb3afd"
server_key_iv: "16591c374fc58505ee76f7ac4b171e55"
server_key_version: "encryptionkey@0"
I20260927 10:43:58.438381 4669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:58.456405 4669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:58.459599 4669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:58.459808 4669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:58.460412 4669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:43:58.461256 4669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:58.461326 4669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:58.461395 4669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:58.461437 4669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:58.470603 4669 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43075
I20260927 10:43:58.470641 4796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43075 every 8 connection(s)
I20260927 10:43:58.471333 4669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:43:58.474269 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 4669
I20260927 10:43:58.474400 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:43:58.474726 392 external_mini_cluster.cc:1428] Setting key a47469100e3870ca30862ee7ba9110d7
Sep 27 10:43:58 dist-test-slave-dzhw krb5kdc[4430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 10:43:58.488039 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:58.473141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35191 (local address 127.0.98.62:35051)
0927 10:43:58.473285 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:58.473288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:58.474215 (+ 927us) server_negotiation.cc:407] Connection header received
0927 10:43:58.475967 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:58.475971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:58.476019 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:58.476127 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:58.477965 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.478779 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.480764 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.481022 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.481112 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:58.484270 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:58.484273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:58.484275 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:58.484301 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:58.485770 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:58.486316 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:58.486320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:58.486323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:58.486402 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:58.486807 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:58.486812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:58.486814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:58.487009 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:58.487142 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:58.487727 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 10:43:58.487862 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:43:58.489022 4799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:58.473510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35051 (local address 127.0.98.2:35191)
0927 10:43:58.473975 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:58.474075 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:58.475751 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:58.476352 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:58.476363 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:58.476910 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:58.477645 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:58.477669 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.478942 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.478949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:58.480589 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:58.480604 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.481199 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.481206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:58.481457 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:58.482181 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:58.482208 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:58.484066 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:58.485900 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:58.485906 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:58.485923 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:58.486224 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:58.486551 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:58.486554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:58.486556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:58.486713 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:58.487140 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:58.487147 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:58.487439 (+ 292us) client_negotiation.cc:769] Sending connection context
0927 10:43:58.487641 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 10:43:58.487999 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":109,"threads_started":1}
I20260927 10:43:58.490391 4797 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35051
I20260927 10:43:58.490684 4797 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:58.491158 4797 heartbeater.cc:511] Master 127.0.98.62:35051 requested a full tablet report, sending...
I20260927 10:43:58.492301 4473 ts_manager.cc:194] Registered new tserver with Master: ee39f7e908104bcb82b9a26cf24c5046 (127.0.98.2:43075)
I20260927 10:43:58.492904 4473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35191
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 10:43:58.512738 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35051
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:43:58.624142 4802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:58.624411 4802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:58.624502 4802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:58.628738 4802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:58.628808 4802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:58.628870 4802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:43:58.634272 4802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:58.635363 4802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:58.636336 4802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:58.646099 4810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:58.646116 4808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:58.646143 4807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:58.646445 4802 server_base.cc:1034] running on GCE node
I20260927 10:43:58.646930 4802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:58.647634 4802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:58.648844 4802 hybrid_clock.cc:648] HybridClock initialized: now 1790505838648826 us; error 40 us; skew 500 ppm
Sep 27 10:43:58 dist-test-slave-dzhw krb5kdc[4430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:43:58.651644 4802 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 10:43:58.652427 4802 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:58.652521 4802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:58.652751 4802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:58.654918 4802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "ba31a499cf994d149104b62c56e4a9bb"
format_stamp: "Formatted at 2026-09-27 10:43:58 on dist-test-slave-dzhw"
server_key: "66aa23fc2b4251210e7fcb12ef1617d3"
server_key_iv: "f03fe5ea6862899217b0842e38d16a13"
server_key_version: "encryptionkey@0"
I20260927 10:43:58.655503 4802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ba31a499cf994d149104b62c56e4a9bb"
format_stamp: "Formatted at 2026-09-27 10:43:58 on dist-test-slave-dzhw"
server_key: "66aa23fc2b4251210e7fcb12ef1617d3"
server_key_iv: "f03fe5ea6862899217b0842e38d16a13"
server_key_version: "encryptionkey@0"
I20260927 10:43:58.659251 4802 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:43:58.661669 4816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:58.663071 4802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:43:58.663215 4802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "ba31a499cf994d149104b62c56e4a9bb"
format_stamp: "Formatted at 2026-09-27 10:43:58 on dist-test-slave-dzhw"
server_key: "66aa23fc2b4251210e7fcb12ef1617d3"
server_key_iv: "f03fe5ea6862899217b0842e38d16a13"
server_key_version: "encryptionkey@0"
I20260927 10:43:58.663334 4802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:58.697244 4802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:58.700508 4802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:58.700731 4802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:58.701332 4802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:43:58.702191 4802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:58.702265 4802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:58.702334 4802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:58.702382 4802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:58.711784 4802 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34447
I20260927 10:43:58.711826 4929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34447 every 8 connection(s)
I20260927 10:43:58.712430 4802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:43:58.719673 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 4802
I20260927 10:43:58.719805 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:43:58.720103 392 external_mini_cluster.cc:1428] Setting key 4c8009d601687b0b2455e138c53c3df9
Sep 27 10:43:58 dist-test-slave-dzhw krb5kdc[4430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:43:58.726265 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:58.714242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48419 (local address 127.0.98.62:35051)
0927 10:43:58.714389 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:58.714392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:58.715357 (+ 965us) server_negotiation.cc:407] Connection header received
0927 10:43:58.716681 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:58.716684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:58.716743 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:58.716821 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:58.717976 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.718544 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.719645 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.719999 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.720119 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:58.723026 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:58.723029 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:58.723031 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:58.723055 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:58.724521 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:58.724964 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:58.724966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:58.724968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:58.725036 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:58.725316 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:58.725318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:58.725319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:58.725480 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:58.725551 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:58.726035 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 10:43:58.726140 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:43:58.727088 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:58.714756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35051 (local address 127.0.98.3:48419)
0927 10:43:58.715202 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:58.715247 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:58.716470 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:58.716984 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:58.716991 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:58.717357 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:58.717825 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:58.717838 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.718652 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.718654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:58.719515 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:58.719523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.720180 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.720187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:58.720360 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:58.721029 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:43:58.721055 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:58.722873 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:58.724617 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:58.724621 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:58.724632 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:58.724883 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:58.725113 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:43:58.725115 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:43:58.725116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:43:58.725240 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:43:58.725562 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:58.725567 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:58.725804 (+ 237us) client_negotiation.cc:769] Sending connection context
0927 10:43:58.725959 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 10:43:58.726204 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":107,"threads_started":1}
I20260927 10:43:58.728339 4930 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35051
I20260927 10:43:58.728560 4930 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:58.728964 4930 heartbeater.cc:511] Master 127.0.98.62:35051 requested a full tablet report, sending...
I20260927 10:43:58.730058 4473 ts_manager.cc:194] Registered new tserver with Master: ba31a499cf994d149104b62c56e4a9bb (127.0.98.3:34447)
I20260927 10:43:58.730703 4473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48419
I20260927 10:43:58.734231 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:43:58.743741 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:58.736136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43510 (local address 127.0.98.62:35051)
0927 10:43:58.736275 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:58.736279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:58.736436 (+ 157us) server_negotiation.cc:407] Connection header received
0927 10:43:58.736534 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:58.736536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:58.736580 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:58.736649 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:58.737401 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.737896 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.738837 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.739038 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.739118 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:58.740492 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:58.740495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:58.740497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:43:58.740518 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:58.742065 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:58.742511 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:58.742513 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:58.742515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:58.742586 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:43:58.742891 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:43:58.742893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:43:58.742894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:43:58.743097 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:58.743201 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:58.743447 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 10:43:58.743539 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:43:58.746924 4473 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:43:58.749945 4473 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:43:58.764683 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:58.759024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34879 (local address 127.0.0.1:46346)
0927 10:43:58.759673 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:58.759731 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:58.759868 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:58.760554 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:58.760560 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:58.760582 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:43:58.760745 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:58.760752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.761839 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.761844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:58.763092 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:58.763102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.764092 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.764096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:58.764207 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:58.764242 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 10:43:58.764394 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 10:43:58.764517 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"spinlock_wait_cycles":1280,"thread_start_us":94,"threads_started":1}
I20260927 10:43:58.764688 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:58.759635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46346 (local address 127.0.98.1:34879)
0927 10:43:58.760006 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:58.760016 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:58.760042 (+ 26us) server_negotiation.cc:407] Connection header received
0927 10:43:58.760140 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:58.760145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:58.760372 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:58.760555 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:43:58.760870 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.761704 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.763232 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.763969 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.764095 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:58.764246 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:58.764428 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 10:43:58.764522 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":88,"threads_started":1}
I20260927 10:43:58.765017 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:58.758766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34447 (local address 127.0.0.1:37896)
0927 10:43:58.759673 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:58.759731 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:58.759868 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:58.760613 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:58.760617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:58.760634 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:43:58.760754 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:58.760761 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.762391 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.762395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:58.763605 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:58.763614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.764642 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.764646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:58.764759 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:58.764779 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:43:58.764829 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 10:43:58.764913 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":65,"spinlock_wait_cycles":7424,"thread_start_us":128,"threads_started":1}
I20260927 10:43:58.765482 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:58.759136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43075 (local address 127.0.0.1:55206)
0927 10:43:58.760154 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:58.760166 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:58.760241 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:58.760902 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:58.760907 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:58.760926 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:43:58.761009 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:58.761014 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.762474 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.762477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:58.763540 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:58.763549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.764642 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.764646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:58.765149 (+ 503us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:58.765176 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:43:58.765259 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 10:43:58.765332 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"mutex_wait_us":425,"thread_start_us":81,"threads_started":1}
I20260927 10:43:58.765566 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:58.759957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55206 (local address 127.0.98.2:43075)
0927 10:43:58.760477 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:58.760488 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:58.760511 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:43:58.760586 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:58.760593 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:58.760755 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:58.760931 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:43:58.761125 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.762239 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.763668 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.764497 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.764691 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:58.764836 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:58.765312 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 10:43:58.765421 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":100,"threads_started":1}
I20260927 10:43:58.766332 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:58.758936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37896 (local address 127.0.98.3:34447)
0927 10:43:58.760129 (+ 1193us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:58.760140 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:58.760163 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:43:58.760241 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:58.760246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:58.760458 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:58.760902 (+ 444us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:43:58.761060 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.762231 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.763715 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.764497 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.765706 (+ 1209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:58.765862 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:58.765915 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 10:43:58.766142 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1090,"thread_start_us":92,"threads_started":1}
I20260927 10:43:58.768616 4598 tablet_service.cc:1467] Processing CreateTablet for tablet 72c913ac772b4ecfb0fbad5521c0ffb2 (DEFAULT_TABLE table=test-table [id=17f37a0a4e3d4d28bc58cc69ead8512c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:58.768616 4731 tablet_service.cc:1467] Processing CreateTablet for tablet 72c913ac772b4ecfb0fbad5521c0ffb2 (DEFAULT_TABLE table=test-table [id=17f37a0a4e3d4d28bc58cc69ead8512c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:58.768826 4863 tablet_service.cc:1467] Processing CreateTablet for tablet 72c913ac772b4ecfb0fbad5521c0ffb2 (DEFAULT_TABLE table=test-table [id=17f37a0a4e3d4d28bc58cc69ead8512c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:58.769732 4863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72c913ac772b4ecfb0fbad5521c0ffb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:58.769742 4731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72c913ac772b4ecfb0fbad5521c0ffb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:58.769742 4598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72c913ac772b4ecfb0fbad5521c0ffb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:58.776527 4947 tablet_bootstrap.cc:492] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ba31a499cf994d149104b62c56e4a9bb: Bootstrap starting.
I20260927 10:43:58.777904 4949 tablet_bootstrap.cc:492] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046: Bootstrap starting.
I20260927 10:43:58.777904 4948 tablet_bootstrap.cc:492] T 72c913ac772b4ecfb0fbad5521c0ffb2 P 81297263c3754576bdfbb5e9a0b7b522: Bootstrap starting.
I20260927 10:43:58.778930 4947 tablet_bootstrap.cc:654] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ba31a499cf994d149104b62c56e4a9bb: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:58.780010 4947 log.cc:824] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ba31a499cf994d149104b62c56e4a9bb: Log is configured to *not* fsync() on all Append() calls
I20260927 10:43:58.780084 4949 tablet_bootstrap.cc:654] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:58.780085 4948 tablet_bootstrap.cc:654] T 72c913ac772b4ecfb0fbad5521c0ffb2 P 81297263c3754576bdfbb5e9a0b7b522: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:58.780964 4948 log.cc:824] T 72c913ac772b4ecfb0fbad5521c0ffb2 P 81297263c3754576bdfbb5e9a0b7b522: Log is configured to *not* fsync() on all Append() calls
I20260927 10:43:58.781185 4949 log.cc:824] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046: Log is configured to *not* fsync() on all Append() calls
I20260927 10:43:58.782673 4947 tablet_bootstrap.cc:492] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ba31a499cf994d149104b62c56e4a9bb: No bootstrap required, opened a new log
I20260927 10:43:58.782888 4947 ts_tablet_manager.cc:1397] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ba31a499cf994d149104b62c56e4a9bb: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 10:43:58.782904 4948 tablet_bootstrap.cc:492] T 72c913ac772b4ecfb0fbad5521c0ffb2 P 81297263c3754576bdfbb5e9a0b7b522: No bootstrap required, opened a new log
I20260927 10:43:58.783025 4949 tablet_bootstrap.cc:492] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046: No bootstrap required, opened a new log
I20260927 10:43:58.783071 4948 ts_tablet_manager.cc:1397] T 72c913ac772b4ecfb0fbad5521c0ffb2 P 81297263c3754576bdfbb5e9a0b7b522: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:43:58.783179 4949 ts_tablet_manager.cc:1397] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:43:58.786113 4947 raft_consensus.cc:348] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ba31a499cf994d149104b62c56e4a9bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba31a499cf994d149104b62c56e4a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34447 } } peers { permanent_uuid: "81297263c3754576bdfbb5e9a0b7b522" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34879 } } peers { permanent_uuid: "ee39f7e908104bcb82b9a26cf24c5046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43075 } }
I20260927 10:43:58.786406 4947 raft_consensus.cc:374] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ba31a499cf994d149104b62c56e4a9bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:58.786479 4947 raft_consensus.cc:729] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ba31a499cf994d149104b62c56e4a9bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba31a499cf994d149104b62c56e4a9bb, State: Initialized, Role: FOLLOWER
I20260927 10:43:58.786953 4947 consensus_queue.cc:260] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ba31a499cf994d149104b62c56e4a9bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba31a499cf994d149104b62c56e4a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34447 } } peers { permanent_uuid: "81297263c3754576bdfbb5e9a0b7b522" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34879 } } peers { permanent_uuid: "ee39f7e908104bcb82b9a26cf24c5046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43075 } }
I20260927 10:43:58.787144 4949 raft_consensus.cc:348] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba31a499cf994d149104b62c56e4a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34447 } } peers { permanent_uuid: "81297263c3754576bdfbb5e9a0b7b522" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34879 } } peers { permanent_uuid: "ee39f7e908104bcb82b9a26cf24c5046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43075 } }
I20260927 10:43:58.787271 4948 raft_consensus.cc:348] T 72c913ac772b4ecfb0fbad5521c0ffb2 P 81297263c3754576bdfbb5e9a0b7b522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba31a499cf994d149104b62c56e4a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34447 } } peers { permanent_uuid: "81297263c3754576bdfbb5e9a0b7b522" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34879 } } peers { permanent_uuid: "ee39f7e908104bcb82b9a26cf24c5046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43075 } }
I20260927 10:43:58.787452 4949 raft_consensus.cc:374] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:58.787483 4948 raft_consensus.cc:374] T 72c913ac772b4ecfb0fbad5521c0ffb2 P 81297263c3754576bdfbb5e9a0b7b522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:58.787509 4949 raft_consensus.cc:729] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee39f7e908104bcb82b9a26cf24c5046, State: Initialized, Role: FOLLOWER
I20260927 10:43:58.787541 4948 raft_consensus.cc:729] T 72c913ac772b4ecfb0fbad5521c0ffb2 P 81297263c3754576bdfbb5e9a0b7b522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81297263c3754576bdfbb5e9a0b7b522, State: Initialized, Role: FOLLOWER
I20260927 10:43:58.787889 4930 heartbeater.cc:503] Master 127.0.98.62:35051 was elected leader, sending a full tablet report...
I20260927 10:43:58.787940 4948 consensus_queue.cc:260] T 72c913ac772b4ecfb0fbad5521c0ffb2 P 81297263c3754576bdfbb5e9a0b7b522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba31a499cf994d149104b62c56e4a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34447 } } peers { permanent_uuid: "81297263c3754576bdfbb5e9a0b7b522" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34879 } } peers { permanent_uuid: "ee39f7e908104bcb82b9a26cf24c5046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43075 } }
I20260927 10:43:58.787988 4949 consensus_queue.cc:260] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba31a499cf994d149104b62c56e4a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34447 } } peers { permanent_uuid: "81297263c3754576bdfbb5e9a0b7b522" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34879 } } peers { permanent_uuid: "ee39f7e908104bcb82b9a26cf24c5046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43075 } }
I20260927 10:43:58.788975 4948 ts_tablet_manager.cc:1428] T 72c913ac772b4ecfb0fbad5521c0ffb2 P 81297263c3754576bdfbb5e9a0b7b522: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:43:58.788975 4949 ts_tablet_manager.cc:1428] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:43:58.789073 4947 ts_tablet_manager.cc:1428] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ba31a499cf994d149104b62c56e4a9bb: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:43:58.789314 4664 heartbeater.cc:503] Master 127.0.98.62:35051 was elected leader, sending a full tablet report...
I20260927 10:43:58.790192 4797 heartbeater.cc:503] Master 127.0.98.62:35051 was elected leader, sending a full tablet report...
I20260927 10:43:58.894595 4954 raft_consensus.cc:482] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:43:58.894886 4954 raft_consensus.cc:504] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba31a499cf994d149104b62c56e4a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34447 } } peers { permanent_uuid: "81297263c3754576bdfbb5e9a0b7b522" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34879 } } peers { permanent_uuid: "ee39f7e908104bcb82b9a26cf24c5046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43075 } }
I20260927 10:43:58.896181 4954 leader_election.cc:288] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba31a499cf994d149104b62c56e4a9bb (127.0.98.3:34447), 81297263c3754576bdfbb5e9a0b7b522 (127.0.98.1:34879)
I20260927 10:43:58.900111 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:58.896649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52269 (local address 127.0.98.1:34879)
0927 10:43:58.896801 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:58.896806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:58.897083 (+ 277us) server_negotiation.cc:407] Connection header received
0927 10:43:58.897146 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:58.897149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:58.897202 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:58.897294 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:43:58.897576 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.898085 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.899141 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.899542 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.899651 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:58.899720 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:58.899908 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 10:43:58.899978 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 10:43:58.900111 4799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:58.896468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34447 (local address 127.0.98.2:58487)
0927 10:43:58.896663 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:58.896679 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:58.896793 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:58.897112 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:58.897115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:58.897137 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:43:58.897297 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:58.897305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.898085 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.898089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:58.899010 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:58.899019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.899707 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.899710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:58.899817 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:58.899833 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 10:43:58.899929 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 10:43:58.899991 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260927 10:43:58.900134 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:58.896544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58487 (local address 127.0.98.3:34447)
0927 10:43:58.896666 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:58.896673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:58.896746 (+ 73us) server_negotiation.cc:407] Connection header received
0927 10:43:58.896899 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:58.896903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:58.896958 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:58.897051 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:43:58.897415 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.897974 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.899148 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.899558 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.899681 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:58.899742 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:58.899965 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 10:43:58.900033 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:43:58.900125 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:58.896546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34879 (local address 127.0.98.2:52269)
0927 10:43:58.896956 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:58.896970 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:58.897064 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:58.897313 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:58.897315 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:58.897328 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:43:58.897462 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:58.897469 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.898196 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.898198 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:58.899010 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:43:58.899018 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.899671 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.899675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:43:58.899775 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:58.899808 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 10:43:58.899868 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 10:43:58.899957 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":138,"threads_started":1}
I20260927 10:43:58.900840 4884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c913ac772b4ecfb0fbad5521c0ffb2" candidate_uuid: "ee39f7e908104bcb82b9a26cf24c5046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba31a499cf994d149104b62c56e4a9bb" is_pre_election: true
I20260927 10:43:58.900978 4618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c913ac772b4ecfb0fbad5521c0ffb2" candidate_uuid: "ee39f7e908104bcb82b9a26cf24c5046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81297263c3754576bdfbb5e9a0b7b522" is_pre_election: true
I20260927 10:43:58.901173 4884 raft_consensus.cc:2454] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ba31a499cf994d149104b62c56e4a9bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee39f7e908104bcb82b9a26cf24c5046 in term 0.
I20260927 10:43:58.901285 4618 raft_consensus.cc:2454] T 72c913ac772b4ecfb0fbad5521c0ffb2 P 81297263c3754576bdfbb5e9a0b7b522 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee39f7e908104bcb82b9a26cf24c5046 in term 0.
I20260927 10:43:58.901772 4685 leader_election.cc:302] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 81297263c3754576bdfbb5e9a0b7b522, ba31a499cf994d149104b62c56e4a9bb, ee39f7e908104bcb82b9a26cf24c5046; no voters:
I20260927 10:43:58.902060 4954 raft_consensus.cc:2781] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:43:58.902143 4954 raft_consensus.cc:482] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:43:58.902189 4954 raft_consensus.cc:3037] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:58.903272 4954 raft_consensus.cc:504] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba31a499cf994d149104b62c56e4a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34447 } } peers { permanent_uuid: "81297263c3754576bdfbb5e9a0b7b522" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34879 } } peers { permanent_uuid: "ee39f7e908104bcb82b9a26cf24c5046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43075 } }
I20260927 10:43:58.903620 4954 leader_election.cc:288] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046 [CANDIDATE]: Term 1 election: Requested vote from peers ba31a499cf994d149104b62c56e4a9bb (127.0.98.3:34447), 81297263c3754576bdfbb5e9a0b7b522 (127.0.98.1:34879)
I20260927 10:43:58.903954 4884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c913ac772b4ecfb0fbad5521c0ffb2" candidate_uuid: "ee39f7e908104bcb82b9a26cf24c5046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba31a499cf994d149104b62c56e4a9bb"
I20260927 10:43:58.904018 4618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c913ac772b4ecfb0fbad5521c0ffb2" candidate_uuid: "ee39f7e908104bcb82b9a26cf24c5046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81297263c3754576bdfbb5e9a0b7b522"
I20260927 10:43:58.904070 4884 raft_consensus.cc:3037] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ba31a499cf994d149104b62c56e4a9bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:58.904210 4618 raft_consensus.cc:3037] T 72c913ac772b4ecfb0fbad5521c0ffb2 P 81297263c3754576bdfbb5e9a0b7b522 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:58.905136 4884 raft_consensus.cc:2454] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ba31a499cf994d149104b62c56e4a9bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee39f7e908104bcb82b9a26cf24c5046 in term 1.
I20260927 10:43:58.905313 4618 raft_consensus.cc:2454] T 72c913ac772b4ecfb0fbad5521c0ffb2 P 81297263c3754576bdfbb5e9a0b7b522 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee39f7e908104bcb82b9a26cf24c5046 in term 1.
I20260927 10:43:58.905458 4685 leader_election.cc:302] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba31a499cf994d149104b62c56e4a9bb, ee39f7e908104bcb82b9a26cf24c5046; no voters:
I20260927 10:43:58.905639 4954 raft_consensus.cc:2781] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:43:58.905853 4954 raft_consensus.cc:686] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046 [term 1 LEADER]: Becoming Leader. State: Replica: ee39f7e908104bcb82b9a26cf24c5046, State: Running, Role: LEADER
I20260927 10:43:58.906203 4954 consensus_queue.cc:237] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba31a499cf994d149104b62c56e4a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34447 } } peers { permanent_uuid: "81297263c3754576bdfbb5e9a0b7b522" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34879 } } peers { permanent_uuid: "ee39f7e908104bcb82b9a26cf24c5046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43075 } }
I20260927 10:43:58.909541 4473 catalog_manager.cc:5495] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee39f7e908104bcb82b9a26cf24c5046 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ee39f7e908104bcb82b9a26cf24c5046" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba31a499cf994d149104b62c56e4a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81297263c3754576bdfbb5e9a0b7b522" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee39f7e908104bcb82b9a26cf24c5046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43075 } health_report { overall_health: HEALTHY } } }
I20260927 10:43:58.954299 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:58.950786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55220 (local address 127.0.98.2:43075)
0927 10:43:58.950935 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:58.950942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:58.950959 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:43:58.951048 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:58.951051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:58.951104 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:58.951222 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:43:58.951493 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.951998 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.952859 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:43:58.953073 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:43:58.953181 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:43:58.953503 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:43:58.953897 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:43:58.954034 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:58.954073 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 10:43:58.954128 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:43:58.960489 4618 raft_consensus.cc:1261] T 72c913ac772b4ecfb0fbad5521c0ffb2 P 81297263c3754576bdfbb5e9a0b7b522 [term 1 FOLLOWER]: Refusing update from remote peer ee39f7e908104bcb82b9a26cf24c5046: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:43:58.960489 4884 raft_consensus.cc:1261] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ba31a499cf994d149104b62c56e4a9bb [term 1 FOLLOWER]: Refusing update from remote peer ee39f7e908104bcb82b9a26cf24c5046: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:43:58.961556 4954 consensus_queue.cc:1035] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81297263c3754576bdfbb5e9a0b7b522" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:43:58.961854 4957 consensus_queue.cc:1035] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba31a499cf994d149104b62c56e4a9bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 10:43:58.963382 4931 tablet.cc:2369] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ba31a499cf994d149104b62c56e4a9bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:43:58.979216 4470 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:43:58.979421 4470 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:43:58.981747 4470 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b041a4132352423f87e77240e3e0722b: Sending DeleteTablet for 3 replicas of tablet 72c913ac772b4ecfb0fbad5521c0ffb2
I20260927 10:43:58.982558 4863 tablet_service.cc:1514] Processing DeleteTablet for tablet 72c913ac772b4ecfb0fbad5521c0ffb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:43:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37896
I20260927 10:43:58.982611 4598 tablet_service.cc:1514] Processing DeleteTablet for tablet 72c913ac772b4ecfb0fbad5521c0ffb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:43:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46346
I20260927 10:43:58.982695 4731 tablet_service.cc:1514] Processing DeleteTablet for tablet 72c913ac772b4ecfb0fbad5521c0ffb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:43:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55206
I20260927 10:43:58.983194 4970 tablet_replica.cc:330] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ba31a499cf994d149104b62c56e4a9bb: stopping tablet replica
I20260927 10:43:58.983538 4970 raft_consensus.cc:2229] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ba31a499cf994d149104b62c56e4a9bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:43:58.983672 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 4536
I20260927 10:43:58.983862 4970 raft_consensus.cc:2258] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ba31a499cf994d149104b62c56e4a9bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:43:58.984201 4971 tablet_replica.cc:330] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046: stopping tablet replica
I20260927 10:43:58.984483 4971 raft_consensus.cc:2229] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:43:58.984818 4971 raft_consensus.cc:2258] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:43:58.986207 4970 ts_tablet_manager.cc:1905] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ba31a499cf994d149104b62c56e4a9bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:43:58.987123 4971 ts_tablet_manager.cc:1905] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:43:58.988981 4970 ts_tablet_manager.cc:1918] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ba31a499cf994d149104b62c56e4a9bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:43:58.989082 4970 log.cc:1196] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ba31a499cf994d149104b62c56e4a9bb: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/72c913ac772b4ecfb0fbad5521c0ffb2
I20260927 10:43:58.989401 4970 ts_tablet_manager.cc:1939] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ba31a499cf994d149104b62c56e4a9bb: Deleting consensus metadata
I20260927 10:43:58.990829 4971 ts_tablet_manager.cc:1918] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:43:58.990871 4459 catalog_manager.cc:4841] TS ba31a499cf994d149104b62c56e4a9bb (127.0.98.3:34447): tablet 72c913ac772b4ecfb0fbad5521c0ffb2 (table test-table [id=17f37a0a4e3d4d28bc58cc69ead8512c]) successfully deleted
I20260927 10:43:58.991039 4971 log.cc:1196] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505812134190-392-0/minicluster-data/ts-1/wal/wals/72c913ac772b4ecfb0fbad5521c0ffb2
I20260927 10:43:58.991384 4971 ts_tablet_manager.cc:1939] T 72c913ac772b4ecfb0fbad5521c0ffb2 P ee39f7e908104bcb82b9a26cf24c5046: Deleting consensus metadata
I20260927 10:43:58.992556 4460 catalog_manager.cc:4841] TS ee39f7e908104bcb82b9a26cf24c5046 (127.0.98.2:43075): tablet 72c913ac772b4ecfb0fbad5521c0ffb2 (table test-table [id=17f37a0a4e3d4d28bc58cc69ead8512c]) successfully deleted
W20260927 10:43:58.992746 4461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34879 (error 108)
I20260927 10:43:58.993796 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:58.993490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34879 (local address 127.0.0.1:46356)
0927 10:43:58.993622 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:58.993698 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 10:43:58.993913 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 4669
W20260927 10:43:58.994015 4461 catalog_manager.cc:4584] TS 81297263c3754576bdfbb5e9a0b7b522 (127.0.98.1:34879): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 72c913ac772b4ecfb0fbad5521c0ffb2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34879: connect: Connection refused (error 111)
I20260927 10:43:58.999712 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 4802
I20260927 10:43:59.004987 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 4443
2026-09-27T10:43:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2419 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T10:43:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:43:59Z Disabled control of system clock
I20260927 10:43:59.035125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38771
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 10:43:59.144884 4977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:59.145187 4977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:59.145224 4977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:59.149222 4977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:43:59.149299 4977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:59.149315 4977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:59.149333 4977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:43:59.149349 4977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:43:59.154057 4977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38771
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:59.155148 4977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:59.156520 4977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:59.165073 4982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:59.165123 4977 server_base.cc:1034] running on GCE node
W20260927 10:43:59.165017 4983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:59.165017 4985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:59.165792 4977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:59.167107 4977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:59.168282 4977 hybrid_clock.cc:648] HybridClock initialized: now 1790505839168273 us; error 36 us; skew 500 ppm
I20260927 10:43:59.170226 4977 webserver.cc:460] Webserver started at http://127.0.98.62:46433/ using document root <none> and password file <none>
I20260927 10:43:59.170903 4977 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:59.170971 4977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:59.171191 4977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:59.173241 4977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "bd5b693022bc44f89cb046dafd78b5f8"
format_stamp: "Formatted at 2026-09-27 10:43:59 on dist-test-slave-dzhw"
server_key: "1b0370481e2311b2ffb6daeb6478d406"
server_key_iv: "d18ca6ac820384e393abe42a39c31f9c"
server_key_version: "encryptionkey@0"
I20260927 10:43:59.173818 4977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "bd5b693022bc44f89cb046dafd78b5f8"
format_stamp: "Formatted at 2026-09-27 10:43:59 on dist-test-slave-dzhw"
server_key: "1b0370481e2311b2ffb6daeb6478d406"
server_key_iv: "d18ca6ac820384e393abe42a39c31f9c"
server_key_version: "encryptionkey@0"
I20260927 10:43:59.177654 4977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 10:43:59.180166 4991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:59.181656 4977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 10:43:59.181825 4977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "bd5b693022bc44f89cb046dafd78b5f8"
format_stamp: "Formatted at 2026-09-27 10:43:59 on dist-test-slave-dzhw"
server_key: "1b0370481e2311b2ffb6daeb6478d406"
server_key_iv: "d18ca6ac820384e393abe42a39c31f9c"
server_key_version: "encryptionkey@0"
I20260927 10:43:59.181955 4977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:59.187094 4977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:59.187723 4977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:59.187922 4977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:59.194638 4977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38771
I20260927 10:43:59.194671 5043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38771 every 8 connection(s)
I20260927 10:43:59.195590 4977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:43:59.198750 5044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:59.200829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 4977
I20260927 10:43:59.200976 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:43:59.201258 392 external_mini_cluster.cc:1428] Setting key 31295a6234093b98d59cf0c14e52fe2c
I20260927 10:43:59.204648 5044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd5b693022bc44f89cb046dafd78b5f8: Bootstrap starting.
I20260927 10:43:59.206902 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:59.202784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57652 (local address 127.0.98.62:38771)
0927 10:43:59.203459 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:59.203477 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:59.203510 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:43:59.204131 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:59.204147 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:59.204493 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:59.204843 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:59.205609 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:59.205614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:59.205625 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:43:59.205635 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:59.205686 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:43:59.205702 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:59.205850 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:59.205921 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 10:43:59.206154 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":141,"threads_started":1}
I20260927 10:43:59.207180 5044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd5b693022bc44f89cb046dafd78b5f8: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:59.207981 5044 log.cc:824] T 00000000000000000000000000000000 P bd5b693022bc44f89cb046dafd78b5f8: Log is configured to *not* fsync() on all Append() calls
I20260927 10:43:59.209997 5044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd5b693022bc44f89cb046dafd78b5f8: No bootstrap required, opened a new log
I20260927 10:43:59.213958 5044 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd5b693022bc44f89cb046dafd78b5f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd5b693022bc44f89cb046dafd78b5f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38771 } }
I20260927 10:43:59.214213 5044 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd5b693022bc44f89cb046dafd78b5f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:59.214257 5044 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd5b693022bc44f89cb046dafd78b5f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd5b693022bc44f89cb046dafd78b5f8, State: Initialized, Role: FOLLOWER
I20260927 10:43:59.214704 5044 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd5b693022bc44f89cb046dafd78b5f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd5b693022bc44f89cb046dafd78b5f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38771 } }
I20260927 10:43:59.214834 5044 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd5b693022bc44f89cb046dafd78b5f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:43:59.214883 5044 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd5b693022bc44f89cb046dafd78b5f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:43:59.214956 5044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd5b693022bc44f89cb046dafd78b5f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:59.215955 5044 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd5b693022bc44f89cb046dafd78b5f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd5b693022bc44f89cb046dafd78b5f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38771 } }
I20260927 10:43:59.216261 5044 leader_election.cc:302] T 00000000000000000000000000000000 P bd5b693022bc44f89cb046dafd78b5f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd5b693022bc44f89cb046dafd78b5f8; no voters:
I20260927 10:43:59.216536 5044 leader_election.cc:288] T 00000000000000000000000000000000 P bd5b693022bc44f89cb046dafd78b5f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:43:59.216718 5049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd5b693022bc44f89cb046dafd78b5f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:43:59.216950 5049 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd5b693022bc44f89cb046dafd78b5f8 [term 1 LEADER]: Becoming Leader. State: Replica: bd5b693022bc44f89cb046dafd78b5f8, State: Running, Role: LEADER
I20260927 10:43:59.217325 5049 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd5b693022bc44f89cb046dafd78b5f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd5b693022bc44f89cb046dafd78b5f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38771 } }
I20260927 10:43:59.217976 5044 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd5b693022bc44f89cb046dafd78b5f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:43:59.219029 5050 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd5b693022bc44f89cb046dafd78b5f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd5b693022bc44f89cb046dafd78b5f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd5b693022bc44f89cb046dafd78b5f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38771 } } }
I20260927 10:43:59.219162 5050 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd5b693022bc44f89cb046dafd78b5f8 [sys.catalog]: This master's current role is: LEADER
I20260927 10:43:59.219414 5051 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd5b693022bc44f89cb046dafd78b5f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd5b693022bc44f89cb046dafd78b5f8. Latest consensus state: current_term: 1 leader_uuid: "bd5b693022bc44f89cb046dafd78b5f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd5b693022bc44f89cb046dafd78b5f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38771 } } }
I20260927 10:43:59.219503 5051 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd5b693022bc44f89cb046dafd78b5f8 [sys.catalog]: This master's current role is: LEADER
I20260927 10:43:59.219571 5058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:43:59.222723 5058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:43:59.228524 5058 catalog_manager.cc:1347] Generated new cluster ID: a49b68290d81457e817182628f61748d
I20260927 10:43:59.228602 5058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:43:59.238075 5058 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:43:59.238406 5058 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:43:59.246234 5058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd5b693022bc44f89cb046dafd78b5f8: Generated new TSK 0
I20260927 10:43:59.247077 5058 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:43:59.259989 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38771
--builtin_ntp_servers=127.0.98.20:44239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 10:43:59.373756 5068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:59.374065 5068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:59.374140 5068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:59.378221 5068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:59.378311 5068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:59.378407 5068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:43:59.383144 5068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:59.384249 5068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:59.385707 5068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:59.395398 5073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:59.395485 5068 server_base.cc:1034] running on GCE node
W20260927 10:43:59.395444 5076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:59.395624 5074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:59.396013 5068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:59.396663 5068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:59.397850 5068 hybrid_clock.cc:648] HybridClock initialized: now 1790505839397832 us; error 29 us; skew 500 ppm
I20260927 10:43:59.399840 5068 webserver.cc:460] Webserver started at http://127.0.98.1:43101/ using document root <none> and password file <none>
I20260927 10:43:59.400390 5068 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:59.400451 5068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:59.400616 5068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:59.402740 5068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "96279d6723fe43fb97a7ed98def84a45"
format_stamp: "Formatted at 2026-09-27 10:43:59 on dist-test-slave-dzhw"
server_key: "ca1c90b4b88330f3b1bf8f167addda9f"
server_key_iv: "202ef72dc8f6dfa81c24771a85eedff8"
server_key_version: "encryptionkey@0"
I20260927 10:43:59.403242 5068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "96279d6723fe43fb97a7ed98def84a45"
format_stamp: "Formatted at 2026-09-27 10:43:59 on dist-test-slave-dzhw"
server_key: "ca1c90b4b88330f3b1bf8f167addda9f"
server_key_iv: "202ef72dc8f6dfa81c24771a85eedff8"
server_key_version: "encryptionkey@0"
I20260927 10:43:59.406906 5068 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:43:59.409368 5082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:59.410728 5068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:43:59.410867 5068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "96279d6723fe43fb97a7ed98def84a45"
format_stamp: "Formatted at 2026-09-27 10:43:59 on dist-test-slave-dzhw"
server_key: "ca1c90b4b88330f3b1bf8f167addda9f"
server_key_iv: "202ef72dc8f6dfa81c24771a85eedff8"
server_key_version: "encryptionkey@0"
I20260927 10:43:59.410988 5068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:59.415611 5068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:59.416226 5068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:59.416414 5068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:59.416954 5068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:43:59.417850 5068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:59.417923 5068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:59.418032 5068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:59.418074 5068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:59.426921 5068 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34467
I20260927 10:43:59.426985 5195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34467 every 8 connection(s)
I20260927 10:43:59.427881 5068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:43:59.433943 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:59.429701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58659 (local address 127.0.98.62:38771)
0927 10:43:59.429843 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:59.429846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:59.430909 (+ 1063us) server_negotiation.cc:407] Connection header received
0927 10:43:59.432234 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:59.432237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:59.432310 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:59.432416 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:59.433472 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:59.433475 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:59.433478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:43:59.433489 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:59.433509 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:43:59.433525 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:59.433619 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:59.433765 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 10:43:59.433810 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:43:59.434836 5198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:59.430084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38771 (local address 127.0.98.1:58659)
0927 10:43:59.430756 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:59.430803 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:59.432024 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:59.432576 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:59.432582 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:59.432667 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:59.433221 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:43:59.433223 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:59.433262 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:43:59.433328 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:59.433613 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:59.433621 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:59.433628 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 10:43:59.433817 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 10:43:59.434049 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":106,"threads_started":1}
I20260927 10:43:59.436039 5196 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38771
I20260927 10:43:59.436282 5196 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:59.436508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 5068
I20260927 10:43:59.436625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:43:59.436781 5196 heartbeater.cc:511] Master 127.0.98.62:38771 requested a full tablet report, sending...
I20260927 10:43:59.436931 392 external_mini_cluster.cc:1428] Setting key e036ba9e92a91ad99b95a53c50f7f0b5
I20260927 10:43:59.438335 5007 ts_manager.cc:194] Registered new tserver with Master: 96279d6723fe43fb97a7ed98def84a45 (127.0.98.1:34467)
I20260927 10:43:59.438872 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38771
--builtin_ntp_servers=127.0.98.20:44239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 10:43:59.551335 5199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:59.551620 5199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:59.551692 5199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:59.555737 5199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:59.555809 5199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:59.555866 5199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:43:59.560628 5199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:59.561844 5199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:59.563265 5199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:59.572853 5207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:59.572857 5204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:59.572857 5205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:59.572921 5199 server_base.cc:1034] running on GCE node
I20260927 10:43:59.573526 5199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:59.574179 5199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:59.575340 5199 hybrid_clock.cc:648] HybridClock initialized: now 1790505839575323 us; error 27 us; skew 500 ppm
I20260927 10:43:59.577153 5199 webserver.cc:460] Webserver started at http://127.0.98.2:45317/ using document root <none> and password file <none>
I20260927 10:43:59.577696 5199 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:59.577757 5199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:59.577929 5199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:59.580003 5199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "9135915822c949a79ebe192cf05a64eb"
format_stamp: "Formatted at 2026-09-27 10:43:59 on dist-test-slave-dzhw"
server_key: "9b40f17f7ceed917c7b8556fb92146d0"
server_key_iv: "d5347ddc3c0139e06cb3578f2234cded"
server_key_version: "encryptionkey@0"
I20260927 10:43:59.580510 5199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9135915822c949a79ebe192cf05a64eb"
format_stamp: "Formatted at 2026-09-27 10:43:59 on dist-test-slave-dzhw"
server_key: "9b40f17f7ceed917c7b8556fb92146d0"
server_key_iv: "d5347ddc3c0139e06cb3578f2234cded"
server_key_version: "encryptionkey@0"
I20260927 10:43:59.584154 5199 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 10:43:59.586632 5213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:59.587898 5199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:43:59.588018 5199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "9135915822c949a79ebe192cf05a64eb"
format_stamp: "Formatted at 2026-09-27 10:43:59 on dist-test-slave-dzhw"
server_key: "9b40f17f7ceed917c7b8556fb92146d0"
server_key_iv: "d5347ddc3c0139e06cb3578f2234cded"
server_key_version: "encryptionkey@0"
I20260927 10:43:59.588125 5199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:59.592736 5199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:59.593329 5199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:59.593503 5199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:59.594044 5199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:43:59.595026 5199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:59.595075 5199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:59.595136 5199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:59.595175 5199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:59.603761 5199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43375
I20260927 10:43:59.603786 5326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43375 every 8 connection(s)
I20260927 10:43:59.604661 5199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:43:59.605034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 5199
I20260927 10:43:59.605158 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:43:59.605481 392 external_mini_cluster.cc:1428] Setting key b16adb5556c4f33ded927f45930b6cfa
I20260927 10:43:59.607919 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38771
--builtin_ntp_servers=127.0.98.20:44239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 10:43:59.611116 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:59.606592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45689 (local address 127.0.98.62:38771)
0927 10:43:59.606725 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:59.606729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:59.607985 (+ 1256us) server_negotiation.cc:407] Connection header received
0927 10:43:59.609324 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:59.609328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:59.609375 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:59.609472 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:59.610628 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:59.610635 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:59.610641 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:43:59.610657 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:59.610682 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:43:59.610694 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:59.610785 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:59.610883 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 10:43:59.610928 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:43:59.611924 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:59.607112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38771 (local address 127.0.98.2:45689)
0927 10:43:59.607844 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:59.607885 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:59.609069 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:59.609616 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:59.609622 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:59.609708 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:59.610318 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:43:59.610320 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:59.610360 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:43:59.610489 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:59.610784 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:59.610790 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:59.610794 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 10:43:59.610915 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 10:43:59.611098 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":160,"threads_started":1}
I20260927 10:43:59.613077 5327 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38771
I20260927 10:43:59.613313 5327 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:59.613718 5327 heartbeater.cc:511] Master 127.0.98.62:38771 requested a full tablet report, sending...
I20260927 10:43:59.614758 5007 ts_manager.cc:194] Registered new tserver with Master: 9135915822c949a79ebe192cf05a64eb (127.0.98.2:43375)
W20260927 10:43:59.719782 5330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:43:59.720099 5330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:43:59.720173 5330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:43:59.724424 5330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:43:59.724485 5330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:43:59.724573 5330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:43:59.729537 5330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:43:59.730675 5330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:43:59.732033 5330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:43:59.741551 5336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:59.741581 5330 server_base.cc:1034] running on GCE node
W20260927 10:43:59.741531 5335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:43:59.741514 5338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:43:59.742168 5330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:43:59.742843 5330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:43:59.744022 5330 hybrid_clock.cc:648] HybridClock initialized: now 1790505839744012 us; error 30 us; skew 500 ppm
I20260927 10:43:59.745867 5330 webserver.cc:460] Webserver started at http://127.0.98.3:40037/ using document root <none> and password file <none>
I20260927 10:43:59.746551 5330 fs_manager.cc:359] Metadata directory not provided
I20260927 10:43:59.746645 5330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:43:59.746881 5330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:43:59.748870 5330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "234f1c17de374a479b55a560c7ec3551"
format_stamp: "Formatted at 2026-09-27 10:43:59 on dist-test-slave-dzhw"
server_key: "fd2fa5644628ae256a40c5bb0e94dc72"
server_key_iv: "b4eae398ea2dbc3716e60e8c769cb04b"
server_key_version: "encryptionkey@0"
I20260927 10:43:59.749408 5330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "234f1c17de374a479b55a560c7ec3551"
format_stamp: "Formatted at 2026-09-27 10:43:59 on dist-test-slave-dzhw"
server_key: "fd2fa5644628ae256a40c5bb0e94dc72"
server_key_iv: "b4eae398ea2dbc3716e60e8c769cb04b"
server_key_version: "encryptionkey@0"
I20260927 10:43:59.753036 5330 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 10:43:59.755751 5344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:59.756950 5330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:43:59.757093 5330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "234f1c17de374a479b55a560c7ec3551"
format_stamp: "Formatted at 2026-09-27 10:43:59 on dist-test-slave-dzhw"
server_key: "fd2fa5644628ae256a40c5bb0e94dc72"
server_key_iv: "b4eae398ea2dbc3716e60e8c769cb04b"
server_key_version: "encryptionkey@0"
I20260927 10:43:59.757208 5330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:43:59.761827 5330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:43:59.762485 5330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:43:59.762660 5330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:43:59.763195 5330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:43:59.764130 5330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:43:59.764199 5330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:59.764266 5330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:43:59.764317 5330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:43:59.772692 5330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37607
I20260927 10:43:59.772732 5457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37607 every 8 connection(s)
I20260927 10:43:59.773577 5330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:43:59.774161 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 5330
I20260927 10:43:59.774313 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:43:59.774621 392 external_mini_cluster.cc:1428] Setting key d7058f4e6c02840f406aef9124bef658
I20260927 10:43:59.779526 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:59.775524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52915 (local address 127.0.98.62:38771)
0927 10:43:59.775647 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:59.775651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:59.776852 (+ 1201us) server_negotiation.cc:407] Connection header received
0927 10:43:59.778105 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:59.778107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:59.778140 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:59.778203 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:59.779189 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:59.779192 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:59.779194 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:43:59.779204 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:59.779221 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:43:59.779228 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:59.779296 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:59.779392 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 10:43:59.779427 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:43:59.780335 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:59.775985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38771 (local address 127.0.98.3:52915)
0927 10:43:59.776717 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:59.776765 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:59.777934 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:59.778330 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:59.778336 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:59.778467 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:59.778965 (+ 498us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:43:59.778966 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:59.779004 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:43:59.779066 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:59.779284 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:59.779288 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:59.779292 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 10:43:59.779408 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 10:43:59.779585 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":101,"threads_started":1}
I20260927 10:43:59.781363 5458 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38771
I20260927 10:43:59.781577 5458 heartbeater.cc:464] Registering TS with master...
I20260927 10:43:59.781961 5458 heartbeater.cc:511] Master 127.0.98.62:38771 requested a full tablet report, sending...
I20260927 10:43:59.782943 5007 ts_manager.cc:194] Registered new tserver with Master: 234f1c17de374a479b55a560c7ec3551 (127.0.98.3:37607)
I20260927 10:43:59.788307 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:43:59.791661 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:59.790099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57664 (local address 127.0.98.62:38771)
0927 10:43:59.790232 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:59.790235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:59.790387 (+ 152us) server_negotiation.cc:407] Connection header received
0927 10:43:59.790561 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:59.790563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:59.790591 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:59.790657 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:59.791400 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:59.791402 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:59.791404 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:43:59.791412 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:59.791426 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:43:59.791433 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:59.791494 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:59.791535 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 10:43:59.791566 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:43:59.794328 5007 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:43:59.797283 5007 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:43:59.808777 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:59.805708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42846 (local address 127.0.98.1:34467)
0927 10:43:59.806699 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:59.806709 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:59.806732 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:43:59.806800 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:59.806806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:59.806989 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:59.807379 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:59.808183 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:59.808187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:59.808195 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:43:59.808202 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:59.808266 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:43:59.808291 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:59.808407 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:59.808551 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 10:43:59.808623 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":906,"thread_start_us":92,"threads_started":1}
I20260927 10:43:59.808770 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:59.805527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34467 (local address 127.0.0.1:42846)
0927 10:43:59.806347 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:59.806362 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:59.806457 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:59.807128 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:59.807131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:59.807145 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:59.807667 (+ 522us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:43:59.807671 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:59.807695 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:43:59.807779 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:59.808387 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:59.808397 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:59.808405 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 10:43:59.808514 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 10:43:59.808596 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":90,"threads_started":1}
I20260927 10:43:59.809084 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:59.805310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37607 (local address 127.0.0.1:35510)
0927 10:43:59.806124 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:59.806171 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:59.806287 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:59.806951 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:59.806957 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:59.807032 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:59.807802 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:43:59.807805 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:59.807816 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:43:59.807845 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:59.808490 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:59.808496 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:59.808499 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 10:43:59.808560 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 10:43:59.808982 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":390,"thread_start_us":103,"threads_started":1}
I20260927 10:43:59.809280 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:59.806179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39952 (local address 127.0.98.2:43375)
0927 10:43:59.806696 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:59.806709 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:59.806992 (+ 283us) server_negotiation.cc:407] Connection header received
0927 10:43:59.807049 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:59.807056 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:59.807235 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:59.807378 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:59.808295 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:59.808300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:59.808311 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:43:59.808326 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:59.808366 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:43:59.808392 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:59.808481 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:59.808840 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 10:43:59.809113 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":109,"threads_started":1}
I20260927 10:43:59.808777 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:59.806046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43375 (local address 127.0.0.1:39952)
0927 10:43:59.806830 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:59.806842 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:59.806911 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:59.807691 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:59.807695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:59.807713 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:59.808121 (+ 408us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:43:59.808126 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:59.808137 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:43:59.808164 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:59.808467 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:59.808470 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:59.808474 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 10:43:59.808550 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 10:43:59.808590 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":78,"threads_started":1}
I20260927 10:43:59.809909 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:59.805433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35510 (local address 127.0.98.3:37607)
0927 10:43:59.806560 (+ 1127us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:59.806571 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:59.806596 (+ 25us) server_negotiation.cc:407] Connection header received
0927 10:43:59.806649 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:59.806654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:59.806812 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:59.806956 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:59.808224 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:59.808232 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:59.808244 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:43:59.808259 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:59.808298 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:43:59.808335 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:59.808451 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:59.809643 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0927 10:43:59.809713 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1024,"thread_start_us":176,"threads_started":1}
I20260927 10:43:59.812179 5261 tablet_service.cc:1467] Processing CreateTablet for tablet f7a77e2a6a284a4783b7c0cf14e2be49 (DEFAULT_TABLE table=test-table [id=c36411117ad74d2d912baf69da873538]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:59.812179 5130 tablet_service.cc:1467] Processing CreateTablet for tablet f7a77e2a6a284a4783b7c0cf14e2be49 (DEFAULT_TABLE table=test-table [id=c36411117ad74d2d912baf69da873538]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:59.813328 5261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7a77e2a6a284a4783b7c0cf14e2be49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:59.813328 5130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7a77e2a6a284a4783b7c0cf14e2be49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:59.816511 5392 tablet_service.cc:1467] Processing CreateTablet for tablet f7a77e2a6a284a4783b7c0cf14e2be49 (DEFAULT_TABLE table=test-table [id=c36411117ad74d2d912baf69da873538]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:43:59.817620 5392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7a77e2a6a284a4783b7c0cf14e2be49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:43:59.823849 5476 tablet_bootstrap.cc:492] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb: Bootstrap starting.
I20260927 10:43:59.823961 5477 tablet_bootstrap.cc:492] T f7a77e2a6a284a4783b7c0cf14e2be49 P 234f1c17de374a479b55a560c7ec3551: Bootstrap starting.
I20260927 10:43:59.824735 5475 tablet_bootstrap.cc:492] T f7a77e2a6a284a4783b7c0cf14e2be49 P 96279d6723fe43fb97a7ed98def84a45: Bootstrap starting.
I20260927 10:43:59.825989 5477 tablet_bootstrap.cc:654] T f7a77e2a6a284a4783b7c0cf14e2be49 P 234f1c17de374a479b55a560c7ec3551: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:59.826149 5476 tablet_bootstrap.cc:654] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:59.826740 5475 tablet_bootstrap.cc:654] T f7a77e2a6a284a4783b7c0cf14e2be49 P 96279d6723fe43fb97a7ed98def84a45: Neither blocks nor log segments found. Creating new log.
I20260927 10:43:59.827064 5476 log.cc:824] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb: Log is configured to *not* fsync() on all Append() calls
I20260927 10:43:59.827464 5477 log.cc:824] T f7a77e2a6a284a4783b7c0cf14e2be49 P 234f1c17de374a479b55a560c7ec3551: Log is configured to *not* fsync() on all Append() calls
I20260927 10:43:59.827636 5475 log.cc:824] T f7a77e2a6a284a4783b7c0cf14e2be49 P 96279d6723fe43fb97a7ed98def84a45: Log is configured to *not* fsync() on all Append() calls
I20260927 10:43:59.829146 5476 tablet_bootstrap.cc:492] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb: No bootstrap required, opened a new log
I20260927 10:43:59.829349 5476 ts_tablet_manager.cc:1397] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:43:59.829427 5475 tablet_bootstrap.cc:492] T f7a77e2a6a284a4783b7c0cf14e2be49 P 96279d6723fe43fb97a7ed98def84a45: No bootstrap required, opened a new log
I20260927 10:43:59.829630 5475 ts_tablet_manager.cc:1397] T f7a77e2a6a284a4783b7c0cf14e2be49 P 96279d6723fe43fb97a7ed98def84a45: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:43:59.830636 5477 tablet_bootstrap.cc:492] T f7a77e2a6a284a4783b7c0cf14e2be49 P 234f1c17de374a479b55a560c7ec3551: No bootstrap required, opened a new log
I20260927 10:43:59.830837 5477 ts_tablet_manager.cc:1397] T f7a77e2a6a284a4783b7c0cf14e2be49 P 234f1c17de374a479b55a560c7ec3551: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20260927 10:43:59.834136 5475 raft_consensus.cc:348] T f7a77e2a6a284a4783b7c0cf14e2be49 P 96279d6723fe43fb97a7ed98def84a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234f1c17de374a479b55a560c7ec3551" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37607 } } peers { permanent_uuid: "9135915822c949a79ebe192cf05a64eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43375 } } peers { permanent_uuid: "96279d6723fe43fb97a7ed98def84a45" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34467 } }
I20260927 10:43:59.834136 5476 raft_consensus.cc:348] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234f1c17de374a479b55a560c7ec3551" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37607 } } peers { permanent_uuid: "9135915822c949a79ebe192cf05a64eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43375 } } peers { permanent_uuid: "96279d6723fe43fb97a7ed98def84a45" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34467 } }
I20260927 10:43:59.834452 5476 raft_consensus.cc:374] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:59.834452 5475 raft_consensus.cc:374] T f7a77e2a6a284a4783b7c0cf14e2be49 P 96279d6723fe43fb97a7ed98def84a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:59.834513 5475 raft_consensus.cc:729] T f7a77e2a6a284a4783b7c0cf14e2be49 P 96279d6723fe43fb97a7ed98def84a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96279d6723fe43fb97a7ed98def84a45, State: Initialized, Role: FOLLOWER
I20260927 10:43:59.834306 5477 raft_consensus.cc:348] T f7a77e2a6a284a4783b7c0cf14e2be49 P 234f1c17de374a479b55a560c7ec3551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234f1c17de374a479b55a560c7ec3551" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37607 } } peers { permanent_uuid: "9135915822c949a79ebe192cf05a64eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43375 } } peers { permanent_uuid: "96279d6723fe43fb97a7ed98def84a45" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34467 } }
I20260927 10:43:59.834540 5476 raft_consensus.cc:729] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9135915822c949a79ebe192cf05a64eb, State: Initialized, Role: FOLLOWER
I20260927 10:43:59.834587 5477 raft_consensus.cc:374] T f7a77e2a6a284a4783b7c0cf14e2be49 P 234f1c17de374a479b55a560c7ec3551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:43:59.834707 5477 raft_consensus.cc:729] T f7a77e2a6a284a4783b7c0cf14e2be49 P 234f1c17de374a479b55a560c7ec3551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 234f1c17de374a479b55a560c7ec3551, State: Initialized, Role: FOLLOWER
I20260927 10:43:59.835026 5475 consensus_queue.cc:260] T f7a77e2a6a284a4783b7c0cf14e2be49 P 96279d6723fe43fb97a7ed98def84a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234f1c17de374a479b55a560c7ec3551" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37607 } } peers { permanent_uuid: "9135915822c949a79ebe192cf05a64eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43375 } } peers { permanent_uuid: "96279d6723fe43fb97a7ed98def84a45" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34467 } }
I20260927 10:43:59.835026 5476 consensus_queue.cc:260] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234f1c17de374a479b55a560c7ec3551" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37607 } } peers { permanent_uuid: "9135915822c949a79ebe192cf05a64eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43375 } } peers { permanent_uuid: "96279d6723fe43fb97a7ed98def84a45" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34467 } }
I20260927 10:43:59.835208 5477 consensus_queue.cc:260] T f7a77e2a6a284a4783b7c0cf14e2be49 P 234f1c17de374a479b55a560c7ec3551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234f1c17de374a479b55a560c7ec3551" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37607 } } peers { permanent_uuid: "9135915822c949a79ebe192cf05a64eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43375 } } peers { permanent_uuid: "96279d6723fe43fb97a7ed98def84a45" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34467 } }
I20260927 10:43:59.835731 5327 heartbeater.cc:503] Master 127.0.98.62:38771 was elected leader, sending a full tablet report...
I20260927 10:43:59.836081 5476 ts_tablet_manager.cc:1428] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 10:43:59.836081 5475 ts_tablet_manager.cc:1428] T f7a77e2a6a284a4783b7c0cf14e2be49 P 96279d6723fe43fb97a7ed98def84a45: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:43:59.836117 5477 ts_tablet_manager.cc:1428] T f7a77e2a6a284a4783b7c0cf14e2be49 P 234f1c17de374a479b55a560c7ec3551: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:43:59.836392 5196 heartbeater.cc:503] Master 127.0.98.62:38771 was elected leader, sending a full tablet report...
I20260927 10:43:59.836448 5458 heartbeater.cc:503] Master 127.0.98.62:38771 was elected leader, sending a full tablet report...
W20260927 10:43:59.855708 5328 tablet.cc:2369] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:43:59.929051 5197 tablet.cc:2369] T f7a77e2a6a284a4783b7c0cf14e2be49 P 96279d6723fe43fb97a7ed98def84a45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:43:59.959673 5481 raft_consensus.cc:482] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:43:59.959900 5481 raft_consensus.cc:504] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234f1c17de374a479b55a560c7ec3551" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37607 } } peers { permanent_uuid: "9135915822c949a79ebe192cf05a64eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43375 } } peers { permanent_uuid: "96279d6723fe43fb97a7ed98def84a45" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34467 } }
I20260927 10:43:59.961160 5481 leader_election.cc:288] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 234f1c17de374a479b55a560c7ec3551 (127.0.98.3:37607), 96279d6723fe43fb97a7ed98def84a45 (127.0.98.1:34467)
I20260927 10:43:59.963212 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:59.961432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37607 (local address 127.0.98.2:36715)
0927 10:43:59.961579 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:59.961597 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:59.961685 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:59.962028 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:59.962031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:59.962047 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:59.962667 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:43:59.962669 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:59.962679 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:43:59.962703 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:59.962951 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:59.962957 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:59.962963 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:43:59.963044 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 10:43:59.963085 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 10:43:59.963254 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:59.961526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36715 (local address 127.0.98.3:37607)
0927 10:43:59.961651 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:59.961658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:59.961675 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:43:59.961841 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:59.961844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:59.961886 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:59.961978 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:59.962809 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:59.962813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:59.962816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:43:59.962827 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:59.962855 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:43:59.962863 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:59.962937 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:59.963058 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 10:43:59.963100 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:43:59.963491 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:59.961522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46667 (local address 127.0.98.1:34467)
0927 10:43:59.961641 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:59.961647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:59.962059 (+ 412us) server_negotiation.cc:407] Connection header received
0927 10:43:59.962219 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:59.962222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:59.962264 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:59.962334 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:59.963073 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:59.963076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:59.963079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:43:59.963090 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:59.963109 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:43:59.963118 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:59.963205 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:59.963347 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 10:43:59.963401 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 10:43:59.963832 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:59.961421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34467 (local address 127.0.98.2:46667)
0927 10:43:59.961976 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0927 10:43:59.961991 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:43:59.962119 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:43:59.962384 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:43:59.962387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:43:59.962402 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:43:59.962880 (+ 478us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:43:59.962881 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:43:59.962888 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:43:59.962929 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:43:59.963212 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:43:59.963215 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:43:59.963220 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 10:43:59.963657 (+ 437us) client_negotiation.cc:240] Negotiation successful
0927 10:43:59.963696 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":148,"thread_start_us":126,"threads_started":1}
I20260927 10:43:59.963982 5412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7a77e2a6a284a4783b7c0cf14e2be49" candidate_uuid: "9135915822c949a79ebe192cf05a64eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "234f1c17de374a479b55a560c7ec3551" is_pre_election: true
I20260927 10:43:59.964324 5412 raft_consensus.cc:2454] T f7a77e2a6a284a4783b7c0cf14e2be49 P 234f1c17de374a479b55a560c7ec3551 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9135915822c949a79ebe192cf05a64eb in term 0.
I20260927 10:43:59.964429 5150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7a77e2a6a284a4783b7c0cf14e2be49" candidate_uuid: "9135915822c949a79ebe192cf05a64eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96279d6723fe43fb97a7ed98def84a45" is_pre_election: true
I20260927 10:43:59.964730 5150 raft_consensus.cc:2454] T f7a77e2a6a284a4783b7c0cf14e2be49 P 96279d6723fe43fb97a7ed98def84a45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9135915822c949a79ebe192cf05a64eb in term 0.
I20260927 10:43:59.964833 5214 leader_election.cc:302] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 234f1c17de374a479b55a560c7ec3551, 9135915822c949a79ebe192cf05a64eb; no voters:
I20260927 10:43:59.965122 5481 raft_consensus.cc:2781] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:43:59.965210 5481 raft_consensus.cc:482] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:43:59.965261 5481 raft_consensus.cc:3037] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:59.966295 5481 raft_consensus.cc:504] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234f1c17de374a479b55a560c7ec3551" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37607 } } peers { permanent_uuid: "9135915822c949a79ebe192cf05a64eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43375 } } peers { permanent_uuid: "96279d6723fe43fb97a7ed98def84a45" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34467 } }
I20260927 10:43:59.966678 5481 leader_election.cc:288] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb [CANDIDATE]: Term 1 election: Requested vote from peers 234f1c17de374a479b55a560c7ec3551 (127.0.98.3:37607), 96279d6723fe43fb97a7ed98def84a45 (127.0.98.1:34467)
I20260927 10:43:59.967056 5412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7a77e2a6a284a4783b7c0cf14e2be49" candidate_uuid: "9135915822c949a79ebe192cf05a64eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "234f1c17de374a479b55a560c7ec3551"
I20260927 10:43:59.967199 5412 raft_consensus.cc:3037] T f7a77e2a6a284a4783b7c0cf14e2be49 P 234f1c17de374a479b55a560c7ec3551 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:59.967238 5150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7a77e2a6a284a4783b7c0cf14e2be49" candidate_uuid: "9135915822c949a79ebe192cf05a64eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96279d6723fe43fb97a7ed98def84a45"
I20260927 10:43:59.967386 5150 raft_consensus.cc:3037] T f7a77e2a6a284a4783b7c0cf14e2be49 P 96279d6723fe43fb97a7ed98def84a45 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:43:59.968288 5412 raft_consensus.cc:2454] T f7a77e2a6a284a4783b7c0cf14e2be49 P 234f1c17de374a479b55a560c7ec3551 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9135915822c949a79ebe192cf05a64eb in term 1.
I20260927 10:43:59.968451 5150 raft_consensus.cc:2454] T f7a77e2a6a284a4783b7c0cf14e2be49 P 96279d6723fe43fb97a7ed98def84a45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9135915822c949a79ebe192cf05a64eb in term 1.
I20260927 10:43:59.968609 5214 leader_election.cc:302] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 234f1c17de374a479b55a560c7ec3551, 9135915822c949a79ebe192cf05a64eb; no voters:
I20260927 10:43:59.968775 5481 raft_consensus.cc:2781] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:43:59.969025 5481 raft_consensus.cc:686] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb [term 1 LEADER]: Becoming Leader. State: Replica: 9135915822c949a79ebe192cf05a64eb, State: Running, Role: LEADER
I20260927 10:43:59.969386 5481 consensus_queue.cc:237] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234f1c17de374a479b55a560c7ec3551" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37607 } } peers { permanent_uuid: "9135915822c949a79ebe192cf05a64eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43375 } } peers { permanent_uuid: "96279d6723fe43fb97a7ed98def84a45" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34467 } }
I20260927 10:43:59.972601 5006 catalog_manager.cc:5495] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 9135915822c949a79ebe192cf05a64eb (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9135915822c949a79ebe192cf05a64eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "234f1c17de374a479b55a560c7ec3551" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9135915822c949a79ebe192cf05a64eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96279d6723fe43fb97a7ed98def84a45" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34467 } health_report { overall_health: UNKNOWN } } }
I20260927 10:43:59.998570 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:43:59.996952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39956 (local address 127.0.98.2:43375)
0927 10:43:59.997104 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 10:43:59.997107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:43:59.997121 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:43:59.997197 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:43:59.997199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:43:59.997256 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:43:59.997331 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:43:59.998088 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:43:59.998091 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:43:59.998093 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:43:59.998104 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:43:59.998122 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:43:59.998131 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:43:59.998213 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 10:43:59.998315 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 10:43:59.998382 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 10:44:00.004738 5412 raft_consensus.cc:1261] T f7a77e2a6a284a4783b7c0cf14e2be49 P 234f1c17de374a479b55a560c7ec3551 [term 1 FOLLOWER]: Refusing update from remote peer 9135915822c949a79ebe192cf05a64eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:00.004738 5150 raft_consensus.cc:1261] T f7a77e2a6a284a4783b7c0cf14e2be49 P 96279d6723fe43fb97a7ed98def84a45 [term 1 FOLLOWER]: Refusing update from remote peer 9135915822c949a79ebe192cf05a64eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:00.005580 5485 consensus_queue.cc:1035] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "96279d6723fe43fb97a7ed98def84a45" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:00.005772 5481 consensus_queue.cc:1035] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "234f1c17de374a479b55a560c7ec3551" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:00.014236 5490 mvcc.cc:205] Tried to move back new op lower bound from 7333911920651788288 to 7333911920517255168. Current Snapshot: MvccSnapshot[applied={T|T < 7333911920651788288}]
I20260927 10:44:00.022745 5007 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:00.022965 5007 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:00.025817 5007 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd5b693022bc44f89cb046dafd78b5f8: Sending DeleteTablet for 3 replicas of tablet f7a77e2a6a284a4783b7c0cf14e2be49
I20260927 10:44:00.026795 5130 tablet_service.cc:1514] Processing DeleteTablet for tablet f7a77e2a6a284a4783b7c0cf14e2be49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:00 UTC) from {username='slave'} at 127.0.0.1:42846
I20260927 10:44:00.026986 5261 tablet_service.cc:1514] Processing DeleteTablet for tablet f7a77e2a6a284a4783b7c0cf14e2be49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:00 UTC) from {username='slave'} at 127.0.0.1:39952
I20260927 10:44:00.026975 5392 tablet_service.cc:1514] Processing DeleteTablet for tablet f7a77e2a6a284a4783b7c0cf14e2be49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:00 UTC) from {username='slave'} at 127.0.0.1:35510
I20260927 10:44:00.027717 5500 tablet_replica.cc:330] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb: stopping tablet replica
I20260927 10:44:00.027803 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 5068
I20260927 10:44:00.029824 5500 raft_consensus.cc:2229] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:44:00.030311 5500 raft_consensus.cc:2258] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:00.030783 5499 tablet_replica.cc:330] T f7a77e2a6a284a4783b7c0cf14e2be49 P 234f1c17de374a479b55a560c7ec3551: stopping tablet replica
I20260927 10:44:00.031078 5499 raft_consensus.cc:2229] T f7a77e2a6a284a4783b7c0cf14e2be49 P 234f1c17de374a479b55a560c7ec3551 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:00.031376 5499 raft_consensus.cc:2258] T f7a77e2a6a284a4783b7c0cf14e2be49 P 234f1c17de374a479b55a560c7ec3551 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:00.032099 5500 ts_tablet_manager.cc:1905] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:00.032720 5499 ts_tablet_manager.cc:1905] T f7a77e2a6a284a4783b7c0cf14e2be49 P 234f1c17de374a479b55a560c7ec3551: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:00.034940 5500 ts_tablet_manager.cc:1918] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:44:00.035117 5500 log.cc:1196] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-1/wal/wals/f7a77e2a6a284a4783b7c0cf14e2be49
I20260927 10:44:00.035523 5500 ts_tablet_manager.cc:1939] T f7a77e2a6a284a4783b7c0cf14e2be49 P 9135915822c949a79ebe192cf05a64eb: Deleting consensus metadata
I20260927 10:44:00.036185 5499 ts_tablet_manager.cc:1918] T f7a77e2a6a284a4783b7c0cf14e2be49 P 234f1c17de374a479b55a560c7ec3551: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:44:00.036358 5499 log.cc:1196] T f7a77e2a6a284a4783b7c0cf14e2be49 P 234f1c17de374a479b55a560c7ec3551: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/f7a77e2a6a284a4783b7c0cf14e2be49
I20260927 10:44:00.036747 5499 ts_tablet_manager.cc:1939] T f7a77e2a6a284a4783b7c0cf14e2be49 P 234f1c17de374a479b55a560c7ec3551: Deleting consensus metadata
I20260927 10:44:00.036968 4992 catalog_manager.cc:4841] TS 9135915822c949a79ebe192cf05a64eb (127.0.98.2:43375): tablet f7a77e2a6a284a4783b7c0cf14e2be49 (table test-table [id=c36411117ad74d2d912baf69da873538]) successfully deleted
W20260927 10:44:00.037570 4993 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34467 (error 108)
I20260927 10:44:00.038018 4992 catalog_manager.cc:4841] TS 234f1c17de374a479b55a560c7ec3551 (127.0.98.3:37607): tablet f7a77e2a6a284a4783b7c0cf14e2be49 (table test-table [id=c36411117ad74d2d912baf69da873538]) successfully deleted
I20260927 10:44:00.038667 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:00.038335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34467 (local address 127.0.0.1:42854)
0927 10:44:00.038493 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:00.038570 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 10:44:00.038727 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 5199
W20260927 10:44:00.038897 4993 catalog_manager.cc:4584] TS 96279d6723fe43fb97a7ed98def84a45 (127.0.98.1:34467): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f7a77e2a6a284a4783b7c0cf14e2be49: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34467: connect: Connection refused (error 111)
I20260927 10:44:00.044610 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 5330
I20260927 10:44:00.049429 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 4977
2026-09-27T10:44:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1043 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 10:44:00.071250 392 mini_oidc.cc:152] Starting JWKS server
I20260927 10:44:00.072795 392 webserver.cc:460] Webserver started at https://127.0.0.1:32949/ using document root <none> and password file <none>
I20260927 10:44:00.072986 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 10:44:00.073272 392 webserver.cc:460] Webserver started at http://127.0.0.1:40065/ using document root <none> and password file <none>
2026-09-27T10:44:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:44:00Z Disabled control of system clock
I20260927 10:44:00.081830 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44623
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44623
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32949/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505812134190-392-0/testchainca.cert with env {}
W20260927 10:44:00.195171 5507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:00.195459 5507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:00.195501 5507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:00.200556 5507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:44:00.200634 5507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:00.200649 5507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:00.200666 5507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:44:00.200681 5507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 10:44:00.200733 5507 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 10:44:00.200760 5507 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32949/jwks.json
I20260927 10:44:00.206367 5507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44623
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32949/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505812134190-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:00.207590 5507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:00.208618 5507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:44:00.217372 5507 server_base.cc:1034] running on GCE node
W20260927 10:44:00.217303 5512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:00.217299 5513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:00.217597 5515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:00.218202 5507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:00.219344 5507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:00.220499 5507 hybrid_clock.cc:648] HybridClock initialized: now 1790505840220486 us; error 36 us; skew 500 ppm
I20260927 10:44:00.222386 5507 webserver.cc:460] Webserver started at http://127.0.98.62:41863/ using document root <none> and password file <none>
I20260927 10:44:00.222990 5507 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:00.223054 5507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:00.223212 5507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:00.225181 5507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "9047c47606414856ada5c545945e1374"
format_stamp: "Formatted at 2026-09-27 10:44:00 on dist-test-slave-dzhw"
server_key: "c2dc7e3368f85a8bce64e9da2a65ee2c"
server_key_iv: "4296c45ab8d0dc5ae6ea9aa2cc2afffe"
server_key_version: "encryptionkey@0"
I20260927 10:44:00.225694 5507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "9047c47606414856ada5c545945e1374"
format_stamp: "Formatted at 2026-09-27 10:44:00 on dist-test-slave-dzhw"
server_key: "c2dc7e3368f85a8bce64e9da2a65ee2c"
server_key_iv: "4296c45ab8d0dc5ae6ea9aa2cc2afffe"
server_key_version: "encryptionkey@0"
I20260927 10:44:00.229615 5507 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 10:44:00.232331 5521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:00.233662 5507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:44:00.233793 5507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "9047c47606414856ada5c545945e1374"
format_stamp: "Formatted at 2026-09-27 10:44:00 on dist-test-slave-dzhw"
server_key: "c2dc7e3368f85a8bce64e9da2a65ee2c"
server_key_iv: "4296c45ab8d0dc5ae6ea9aa2cc2afffe"
server_key_version: "encryptionkey@0"
I20260927 10:44:00.233882 5507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:00.283595 5507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:00.284271 5507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:00.284425 5507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:00.291621 5507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44623
I20260927 10:44:00.291646 5573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44623 every 8 connection(s)
I20260927 10:44:00.292588 5507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:44:00.295764 5574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:00.298269 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 5507
I20260927 10:44:00.298458 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:44:00.298754 392 external_mini_cluster.cc:1428] Setting key e8f6541942d270a1e44ec3f0004fc406
I20260927 10:44:00.301584 5574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9047c47606414856ada5c545945e1374: Bootstrap starting.
I20260927 10:44:00.304139 5574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9047c47606414856ada5c545945e1374: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:00.304976 5574 log.cc:824] T 00000000000000000000000000000000 P 9047c47606414856ada5c545945e1374: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:00.307674 5574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9047c47606414856ada5c545945e1374: No bootstrap required, opened a new log
I20260927 10:44:00.308241 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:00.300071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51552 (local address 127.0.98.62:44623)
0927 10:44:00.300714 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:00.300728 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:00.300764 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:44:00.301370 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:00.301385 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:00.301825 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:00.302106 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:00.302530 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:00.303485 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:00.304681 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:00.305198 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:00.305323 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:00.306186 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:00.306191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:00.306205 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:44:00.306660 (+ 455us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:00.306724 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:44:00.306745 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:00.306825 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:00.306991 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 10:44:00.307416 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":138,"threads_started":1}
I20260927 10:44:00.310982 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:44:00.311532 5574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9047c47606414856ada5c545945e1374 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9047c47606414856ada5c545945e1374" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44623 } }
I20260927 10:44:00.311815 5574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9047c47606414856ada5c545945e1374 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:00.311918 5574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9047c47606414856ada5c545945e1374 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9047c47606414856ada5c545945e1374, State: Initialized, Role: FOLLOWER
I20260927 10:44:00.312341 5574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9047c47606414856ada5c545945e1374 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9047c47606414856ada5c545945e1374" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44623 } }
I20260927 10:44:00.312500 5574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9047c47606414856ada5c545945e1374 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:44:00.312597 5574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9047c47606414856ada5c545945e1374 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:44:00.312700 5574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9047c47606414856ada5c545945e1374 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:00.313655 5574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9047c47606414856ada5c545945e1374 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9047c47606414856ada5c545945e1374" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44623 } }
I20260927 10:44:00.313992 5574 leader_election.cc:302] T 00000000000000000000000000000000 P 9047c47606414856ada5c545945e1374 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9047c47606414856ada5c545945e1374; no voters:
I20260927 10:44:00.314296 5574 leader_election.cc:288] T 00000000000000000000000000000000 P 9047c47606414856ada5c545945e1374 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:44:00.314464 5579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9047c47606414856ada5c545945e1374 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:00.314726 5579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9047c47606414856ada5c545945e1374 [term 1 LEADER]: Becoming Leader. State: Replica: 9047c47606414856ada5c545945e1374, State: Running, Role: LEADER
I20260927 10:44:00.315063 5579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9047c47606414856ada5c545945e1374 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9047c47606414856ada5c545945e1374" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44623 } }
I20260927 10:44:00.315595 5574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9047c47606414856ada5c545945e1374 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:44:00.316658 5581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9047c47606414856ada5c545945e1374 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9047c47606414856ada5c545945e1374. Latest consensus state: current_term: 1 leader_uuid: "9047c47606414856ada5c545945e1374" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9047c47606414856ada5c545945e1374" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44623 } } }
I20260927 10:44:00.316798 5581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9047c47606414856ada5c545945e1374 [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:00.317158 5588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:44:00.317117 5580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9047c47606414856ada5c545945e1374 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9047c47606414856ada5c545945e1374" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9047c47606414856ada5c545945e1374" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44623 } } }
I20260927 10:44:00.317281 5580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9047c47606414856ada5c545945e1374 [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:00.320456 5588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:44:00.326160 5588 catalog_manager.cc:1347] Generated new cluster ID: 7a115125e70845afae19803c4aa3aed4
I20260927 10:44:00.326259 5588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:44:00.344095 5588 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:44:00.345394 5588 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:44:00.365346 5588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9047c47606414856ada5c545945e1374: Generated new TSK 0
I20260927 10:44:00.366138 5588 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 10:44:00.403403 5604 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 10:44:00.403532 5604 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 10:44:00.404040 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:00.387283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51556 (local address 127.0.98.62:44623)
0927 10:44:00.387428 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:00.387433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:00.387645 (+ 212us) server_negotiation.cc:407] Connection header received
0927 10:44:00.387863 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:00.387866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:00.387913 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:00.387995 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0927 10:44:00.388301 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:00.388822 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:00.390089 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:00.390320 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:00.390428 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:00.390807 (+ 379us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 10:44:00.403686 (+ 12879us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32949/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 10:44:00.403933 (+ 247us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51556: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32949/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 10:44:00.404142 5577 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51556: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32949/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 10:44:00.405478 5603 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:00.387192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44623 (local address 127.0.0.1:51556)
0927 10:44:00.387553 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:00.387569 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:00.387708 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:00.388039 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:00.388042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:00.388056 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0927 10:44:00.388184 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:00.388190 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:00.388958 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:00.388962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:00.389951 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:00.389961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:00.390442 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:00.390463 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:00.390556 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:00.390608 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 10:44:00.403933 (+ 13325us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32949/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 10:44:00.404178 (+ 245us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44623: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32949/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":118,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:44623: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32949/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 10:44:00.406234 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 5507
2026-09-27T10:44:00Z chronyd exiting
I20260927 10:44:00.443578 392 test_util.cc:175] -----------------------------------------------
I20260927 10:44:00.443682 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505812134190-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (373 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 10:44:00.444703 392 mini_oidc.cc:152] Starting JWKS server
I20260927 10:44:00.445653 392 webserver.cc:460] Webserver started at https://127.0.0.1:36457/ using document root <none> and password file <none>
I20260927 10:44:00.445802 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 10:44:00.446178 392 webserver.cc:460] Webserver started at http://127.0.0.1:44949/ using document root <none> and password file <none>
2026-09-27T10:44:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:44:00Z Disabled control of system clock
I20260927 10:44:00.454742 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45835
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45835
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36457/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505812134190-392-0/testchainca.cert with env {}
W20260927 10:44:00.564700 5610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:00.565052 5610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:00.565136 5610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:00.569578 5610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:44:00.569702 5610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:00.569736 5610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:00.569777 5610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:44:00.569808 5610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 10:44:00.569849 5610 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 10:44:00.569882 5610 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36457/jwks.json
I20260927 10:44:00.575649 5610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45835
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36457/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505812134190-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:00.576804 5610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:00.577836 5610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:00.586339 5618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:00.586339 5616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:00.586357 5615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:00.586537 5610 server_base.cc:1034] running on GCE node
I20260927 10:44:00.587262 5610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:00.588388 5610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:00.589581 5610 hybrid_clock.cc:648] HybridClock initialized: now 1790505840589574 us; error 31 us; skew 500 ppm
I20260927 10:44:00.591619 5610 webserver.cc:460] Webserver started at http://127.0.98.62:33317/ using document root <none> and password file <none>
I20260927 10:44:00.592392 5610 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:00.592504 5610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:00.592729 5610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:00.594827 5610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "3dc38f1148324f0887a386c27112a563"
format_stamp: "Formatted at 2026-09-27 10:44:00 on dist-test-slave-dzhw"
server_key: "f2320e8891194c5a354a2a2ec7746111"
server_key_iv: "f02bdbed58f25415233fb41a129c2e50"
server_key_version: "encryptionkey@0"
I20260927 10:44:00.595391 5610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "3dc38f1148324f0887a386c27112a563"
format_stamp: "Formatted at 2026-09-27 10:44:00 on dist-test-slave-dzhw"
server_key: "f2320e8891194c5a354a2a2ec7746111"
server_key_iv: "f02bdbed58f25415233fb41a129c2e50"
server_key_version: "encryptionkey@0"
I20260927 10:44:00.599197 5610 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 10:44:00.601655 5624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:00.603019 5610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:44:00.603161 5610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "3dc38f1148324f0887a386c27112a563"
format_stamp: "Formatted at 2026-09-27 10:44:00 on dist-test-slave-dzhw"
server_key: "f2320e8891194c5a354a2a2ec7746111"
server_key_iv: "f02bdbed58f25415233fb41a129c2e50"
server_key_version: "encryptionkey@0"
I20260927 10:44:00.603282 5610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:00.642796 5610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:00.643529 5610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:00.643729 5610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:00.650451 5610 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45835
I20260927 10:44:00.650489 5676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45835 every 8 connection(s)
I20260927 10:44:00.651450 5610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:44:00.654652 5677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:00.660316 5677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dc38f1148324f0887a386c27112a563: Bootstrap starting.
I20260927 10:44:00.660948 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 5610
I20260927 10:44:00.661129 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:44:00.661410 392 external_mini_cluster.cc:1428] Setting key d81824a2bb3366701f600004ed5e4b3b
I20260927 10:44:00.663576 5677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dc38f1148324f0887a386c27112a563: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:00.664346 5677 log.cc:824] T 00000000000000000000000000000000 P 3dc38f1148324f0887a386c27112a563: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:00.666260 5677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dc38f1148324f0887a386c27112a563: No bootstrap required, opened a new log
I20260927 10:44:00.669914 5677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dc38f1148324f0887a386c27112a563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc38f1148324f0887a386c27112a563" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } }
I20260927 10:44:00.670209 5677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dc38f1148324f0887a386c27112a563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:00.670284 5677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dc38f1148324f0887a386c27112a563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc38f1148324f0887a386c27112a563, State: Initialized, Role: FOLLOWER
I20260927 10:44:00.670779 5677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dc38f1148324f0887a386c27112a563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc38f1148324f0887a386c27112a563" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } }
I20260927 10:44:00.670907 5677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dc38f1148324f0887a386c27112a563 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:44:00.671016 5677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dc38f1148324f0887a386c27112a563 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:44:00.671128 5677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dc38f1148324f0887a386c27112a563 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:00.671445 5680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:00.663003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49686 (local address 127.0.98.62:45835)
0927 10:44:00.663734 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:00.663750 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:00.663796 (+ 46us) server_negotiation.cc:407] Connection header received
0927 10:44:00.664530 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:00.664550 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:00.665070 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:00.665437 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:00.665865 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:00.666948 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:00.667846 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:00.668318 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:00.668493 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:00.669424 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:00.669429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:00.669442 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:44:00.669840 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:00.669895 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:44:00.669915 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:00.670021 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:00.670166 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 10:44:00.670486 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":139,"threads_started":1}
I20260927 10:44:00.672108 5677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dc38f1148324f0887a386c27112a563 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc38f1148324f0887a386c27112a563" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } }
I20260927 10:44:00.672489 5677 leader_election.cc:302] T 00000000000000000000000000000000 P 3dc38f1148324f0887a386c27112a563 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dc38f1148324f0887a386c27112a563; no voters:
I20260927 10:44:00.672819 5677 leader_election.cc:288] T 00000000000000000000000000000000 P 3dc38f1148324f0887a386c27112a563 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:44:00.672977 5682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dc38f1148324f0887a386c27112a563 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:00.673223 5682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dc38f1148324f0887a386c27112a563 [term 1 LEADER]: Becoming Leader. State: Replica: 3dc38f1148324f0887a386c27112a563, State: Running, Role: LEADER
I20260927 10:44:00.673563 5682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dc38f1148324f0887a386c27112a563 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc38f1148324f0887a386c27112a563" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } }
I20260927 10:44:00.674144 5677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dc38f1148324f0887a386c27112a563 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:44:00.674645 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:44:00.675407 5683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dc38f1148324f0887a386c27112a563 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dc38f1148324f0887a386c27112a563" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc38f1148324f0887a386c27112a563" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } }
I20260927 10:44:00.675417 5684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dc38f1148324f0887a386c27112a563 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dc38f1148324f0887a386c27112a563. Latest consensus state: current_term: 1 leader_uuid: "3dc38f1148324f0887a386c27112a563" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc38f1148324f0887a386c27112a563" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45835 } } }
I20260927 10:44:00.675529 5683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dc38f1148324f0887a386c27112a563 [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:00.675536 5684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dc38f1148324f0887a386c27112a563 [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:00.675879 5690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:44:00.679046 5690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:44:00.684641 5690 catalog_manager.cc:1347] Generated new cluster ID: 6e177ff5fdf1401eb99bdc9ea2af6b84
I20260927 10:44:00.684778 5690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:44:00.698657 5690 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:44:00.699702 5690 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:44:00.705415 5690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dc38f1148324f0887a386c27112a563: Generated new TSK 0
I20260927 10:44:00.706266 5690 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 10:44:00.737500 5707 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 10:44:00.737596 5707 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 10:44:00.738232 5680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:00.721996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49700 (local address 127.0.98.62:45835)
0927 10:44:00.722142 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:00.722146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:00.722346 (+ 200us) server_negotiation.cc:407] Connection header received
0927 10:44:00.722540 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:00.722543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:00.722592 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:00.722681 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0927 10:44:00.722951 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:00.723489 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:00.724737 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:00.724963 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:00.725039 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:00.725365 (+ 326us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 10:44:00.737904 (+ 12539us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36457/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 10:44:00.738124 (+ 220us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49700: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36457/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 10:44:00.738317 5680 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49700: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36457/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 10:44:00.738524 5706 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:00.721914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45835 (local address 127.0.0.1:49700)
0927 10:44:00.722244 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:00.722259 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:00.722385 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:00.722720 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:00.722723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:00.722732 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 10:44:00.722847 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:00.722853 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:00.723612 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:00.723616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:00.724620 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:00.724631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:00.725064 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:00.725086 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:00.725171 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:00.725200 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 10:44:00.738105 (+ 12905us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36457/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 10:44:00.738317 (+ 212us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45835: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36457/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":98,"threads_started":1}
I20260927 10:44:00.739404 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 5610
2026-09-27T10:44:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (333 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 10:44:00.778671 392 mini_oidc.cc:152] Starting JWKS server
I20260927 10:44:00.779523 392 webserver.cc:460] Webserver started at https://127.0.0.1:46283/ using document root <none> and password file <none>
I20260927 10:44:00.779608 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 10:44:00.779820 392 webserver.cc:460] Webserver started at http://127.0.0.1:33319/ using document root <none> and password file <none>
2026-09-27T10:44:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:44:00Z Disabled control of system clock
I20260927 10:44:00.788805 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43441
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43441
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46283/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 10:44:00.902272 5713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:00.902602 5713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:00.902644 5713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:00.906819 5713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:44:00.906898 5713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:00.906915 5713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:00.906932 5713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:44:00.906948 5713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 10:44:00.906965 5713 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 10:44:00.906977 5713 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46283/jwks.json
I20260927 10:44:00.911900 5713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43441
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46283/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:00.913021 5713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:00.914516 5713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:00.923657 5719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:00.923676 5718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:00.923663 5721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:00.923696 5713 server_base.cc:1034] running on GCE node
I20260927 10:44:00.924633 5713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:00.925850 5713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:00.927639 5713 hybrid_clock.cc:648] HybridClock initialized: now 1790505840927613 us; error 37 us; skew 500 ppm
I20260927 10:44:00.930397 5713 webserver.cc:460] Webserver started at http://127.0.98.62:42295/ using document root <none> and password file <none>
I20260927 10:44:00.931320 5713 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:00.931437 5713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:00.931723 5713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:00.934831 5713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "4dba5cd1779a48a69dfbed0c909bd466"
format_stamp: "Formatted at 2026-09-27 10:44:00 on dist-test-slave-dzhw"
server_key: "8d405be1a006faee03c89ed62ebaaf64"
server_key_iv: "621ecfa76c9dd9f7574c186fe2273e12"
server_key_version: "encryptionkey@0"
I20260927 10:44:00.935577 5713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "4dba5cd1779a48a69dfbed0c909bd466"
format_stamp: "Formatted at 2026-09-27 10:44:00 on dist-test-slave-dzhw"
server_key: "8d405be1a006faee03c89ed62ebaaf64"
server_key_iv: "621ecfa76c9dd9f7574c186fe2273e12"
server_key_version: "encryptionkey@0"
I20260927 10:44:00.942201 5713 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 10:44:00.946597 5727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:00.948372 5713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:44:00.948525 5713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "4dba5cd1779a48a69dfbed0c909bd466"
format_stamp: "Formatted at 2026-09-27 10:44:00 on dist-test-slave-dzhw"
server_key: "8d405be1a006faee03c89ed62ebaaf64"
server_key_iv: "621ecfa76c9dd9f7574c186fe2273e12"
server_key_version: "encryptionkey@0"
I20260927 10:44:00.948700 5713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:00.971503 5713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:00.972522 5713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:00.972784 5713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:00.982884 5713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43441
I20260927 10:44:00.982903 5779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43441 every 8 connection(s)
I20260927 10:44:00.984084 5713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:44:00.985118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 5713
I20260927 10:44:00.985235 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:44:00.985514 392 external_mini_cluster.cc:1428] Setting key a76a71cb8a2cd0c429e2b4fc0490854e
I20260927 10:44:00.988665 5780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:00.996002 5780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dba5cd1779a48a69dfbed0c909bd466: Bootstrap starting.
I20260927 10:44:00.996495 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:00.987207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35694 (local address 127.0.98.62:43441)
0927 10:44:00.987913 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:00.987934 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:00.987977 (+ 43us) server_negotiation.cc:407] Connection header received
0927 10:44:00.988887 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:00.988909 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:00.989422 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:00.989930 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:00.990395 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:00.991903 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:00.992823 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:00.993313 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:00.993491 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:00.994670 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:00.994674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:00.994689 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:44:00.995107 (+ 418us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:00.995169 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:44:00.995190 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:00.995304 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:00.995434 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 10:44:00.995701 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":144,"threads_started":1}
I20260927 10:44:00.998556 5780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dba5cd1779a48a69dfbed0c909bd466: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:00.999403 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:44:00.999507 5780 log.cc:824] T 00000000000000000000000000000000 P 4dba5cd1779a48a69dfbed0c909bd466: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:01.001744 5780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dba5cd1779a48a69dfbed0c909bd466: No bootstrap required, opened a new log
I20260927 10:44:01.005343 5780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dba5cd1779a48a69dfbed0c909bd466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dba5cd1779a48a69dfbed0c909bd466" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43441 } }
I20260927 10:44:01.005661 5780 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dba5cd1779a48a69dfbed0c909bd466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:01.005744 5780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dba5cd1779a48a69dfbed0c909bd466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dba5cd1779a48a69dfbed0c909bd466, State: Initialized, Role: FOLLOWER
I20260927 10:44:01.006256 5780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dba5cd1779a48a69dfbed0c909bd466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dba5cd1779a48a69dfbed0c909bd466" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43441 } }
I20260927 10:44:01.006459 5780 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dba5cd1779a48a69dfbed0c909bd466 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:44:01.006524 5780 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dba5cd1779a48a69dfbed0c909bd466 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:44:01.006608 5780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dba5cd1779a48a69dfbed0c909bd466 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:01.007565 5780 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dba5cd1779a48a69dfbed0c909bd466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dba5cd1779a48a69dfbed0c909bd466" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43441 } }
I20260927 10:44:01.007918 5780 leader_election.cc:302] T 00000000000000000000000000000000 P 4dba5cd1779a48a69dfbed0c909bd466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dba5cd1779a48a69dfbed0c909bd466; no voters:
I20260927 10:44:01.008286 5780 leader_election.cc:288] T 00000000000000000000000000000000 P 4dba5cd1779a48a69dfbed0c909bd466 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:44:01.008435 5785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dba5cd1779a48a69dfbed0c909bd466 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:01.008708 5785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dba5cd1779a48a69dfbed0c909bd466 [term 1 LEADER]: Becoming Leader. State: Replica: 4dba5cd1779a48a69dfbed0c909bd466, State: Running, Role: LEADER
I20260927 10:44:01.009023 5785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dba5cd1779a48a69dfbed0c909bd466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dba5cd1779a48a69dfbed0c909bd466" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43441 } }
I20260927 10:44:01.009567 5780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dba5cd1779a48a69dfbed0c909bd466 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:44:01.010823 5786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dba5cd1779a48a69dfbed0c909bd466 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dba5cd1779a48a69dfbed0c909bd466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dba5cd1779a48a69dfbed0c909bd466" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43441 } } }
I20260927 10:44:01.010823 5787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dba5cd1779a48a69dfbed0c909bd466 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dba5cd1779a48a69dfbed0c909bd466. Latest consensus state: current_term: 1 leader_uuid: "4dba5cd1779a48a69dfbed0c909bd466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dba5cd1779a48a69dfbed0c909bd466" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43441 } } }
I20260927 10:44:01.011006 5786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dba5cd1779a48a69dfbed0c909bd466 [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:01.011006 5787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dba5cd1779a48a69dfbed0c909bd466 [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:01.011459 5794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:44:01.014685 5794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:44:01.021095 5794 catalog_manager.cc:1347] Generated new cluster ID: 1d9ed99e49c04a42ac232bc0d9e5aac9
I20260927 10:44:01.021178 5794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:44:01.035029 5794 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:44:01.036000 5794 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:44:01.044565 5794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dba5cd1779a48a69dfbed0c909bd466: Generated new TSK 0
I20260927 10:44:01.045301 5794 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:44:01.059894 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:01.054490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35696 (local address 127.0.98.62:43441)
0927 10:44:01.054631 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:01.054635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:01.054793 (+ 158us) server_negotiation.cc:407] Connection header received
0927 10:44:01.054961 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:01.054963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:01.055011 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:01.055137 (+ 126us) server_negotiation.cc:226] Negotiated authn=JWT
0927 10:44:01.055406 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:01.055986 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:01.057386 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:01.057597 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:01.057672 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:01.058035 (+ 363us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 10:44:01.059518 (+ 1483us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46283/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 10:44:01.059741 (+ 223us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35696: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46283/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 10:44:01.059968 5783 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35696: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46283/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 10:44:01.061877 5809 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:01.054303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43441 (local address 127.0.0.1:35696)
0927 10:44:01.054680 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:01.054699 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:01.054835 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:01.055131 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:01.055135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:01.055144 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 10:44:01.055297 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:01.055304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:01.056113 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:01.056118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:01.057241 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:01.057253 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:01.057702 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:01.057736 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:01.057833 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:01.057865 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 10:44:01.059688 (+ 1823us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46283/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 10:44:01.061661 (+ 1973us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43441: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46283/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":1661,"thread_start_us":138,"threads_started":1}
I20260927 10:44:01.062520 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 5713
E20260927 10:44:01.070891 5810 webserver.cc:533] Webserver: sslize failed:
E20260927 10:44:01.070974 5810 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T10:44:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (323 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T10:44:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:44:01Z Disabled control of system clock
I20260927 10:44:01.111159 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41311
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41311
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505812134190-392-0/testchainca.cert with env {}
W20260927 10:44:01.225481 5814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:01.225785 5814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:01.225827 5814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:01.230095 5814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:44:01.230165 5814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:01.230185 5814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:01.230202 5814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:44:01.230218 5814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 10:44:01.230280 5814 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 10:44:01.230300 5814 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 10:44:01.235965 5814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41311
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505812134190-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:01.237105 5814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:01.238077 5814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:01.246811 5819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:01.246966 5814 server_base.cc:1034] running on GCE node
W20260927 10:44:01.246843 5822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:01.246843 5820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:01.247700 5814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:01.248792 5814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:01.250005 5814 hybrid_clock.cc:648] HybridClock initialized: now 1790505841249987 us; error 33 us; skew 500 ppm
I20260927 10:44:01.252106 5814 webserver.cc:460] Webserver started at http://127.0.98.62:35075/ using document root <none> and password file <none>
I20260927 10:44:01.252710 5814 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:01.252810 5814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:01.253041 5814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:01.255175 5814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "82626ee153644a198ebd16ec6a843bdf"
format_stamp: "Formatted at 2026-09-27 10:44:01 on dist-test-slave-dzhw"
server_key: "5324b8b626694050309f392e23cb9b42"
server_key_iv: "267693ba8693af5797e4a644a19d33ea"
server_key_version: "encryptionkey@0"
I20260927 10:44:01.255762 5814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "82626ee153644a198ebd16ec6a843bdf"
format_stamp: "Formatted at 2026-09-27 10:44:01 on dist-test-slave-dzhw"
server_key: "5324b8b626694050309f392e23cb9b42"
server_key_iv: "267693ba8693af5797e4a644a19d33ea"
server_key_version: "encryptionkey@0"
I20260927 10:44:01.259646 5814 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 10:44:01.262184 5828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:01.263549 5814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:44:01.263748 5814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "82626ee153644a198ebd16ec6a843bdf"
format_stamp: "Formatted at 2026-09-27 10:44:01 on dist-test-slave-dzhw"
server_key: "5324b8b626694050309f392e23cb9b42"
server_key_iv: "267693ba8693af5797e4a644a19d33ea"
server_key_version: "encryptionkey@0"
I20260927 10:44:01.263882 5814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:01.287500 5814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:01.288249 5814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:01.288461 5814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:01.296140 5814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41311
I20260927 10:44:01.296175 5880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41311 every 8 connection(s)
I20260927 10:44:01.297159 5814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:44:01.297513 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 5814
I20260927 10:44:01.297658 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:44:01.298143 392 external_mini_cluster.cc:1428] Setting key 790e929c0c436a7a1ab5130409e1b168
I20260927 10:44:01.301252 5881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:01.307561 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:01.299568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33020 (local address 127.0.98.62:41311)
0927 10:44:01.300232 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:01.300247 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:01.300283 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:44:01.300940 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:01.300956 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:01.301363 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:01.301773 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:01.302247 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:01.303293 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:01.304223 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:01.304700 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:01.304870 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:01.305770 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:01.305775 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:01.305787 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:44:01.306180 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:01.306233 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:44:01.306251 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:01.306368 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:01.306532 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 10:44:01.306792 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":162,"threads_started":1}
I20260927 10:44:01.308702 5881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82626ee153644a198ebd16ec6a843bdf: Bootstrap starting.
I20260927 10:44:01.310685 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:44:01.311280 5881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82626ee153644a198ebd16ec6a843bdf: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:01.312165 5881 log.cc:824] T 00000000000000000000000000000000 P 82626ee153644a198ebd16ec6a843bdf: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:01.314069 5881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82626ee153644a198ebd16ec6a843bdf: No bootstrap required, opened a new log
I20260927 10:44:01.317850 5881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82626ee153644a198ebd16ec6a843bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82626ee153644a198ebd16ec6a843bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41311 } }
I20260927 10:44:01.318136 5881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82626ee153644a198ebd16ec6a843bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:01.318225 5881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82626ee153644a198ebd16ec6a843bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82626ee153644a198ebd16ec6a843bdf, State: Initialized, Role: FOLLOWER
I20260927 10:44:01.318758 5881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82626ee153644a198ebd16ec6a843bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82626ee153644a198ebd16ec6a843bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41311 } }
I20260927 10:44:01.318917 5881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82626ee153644a198ebd16ec6a843bdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:44:01.319008 5881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82626ee153644a198ebd16ec6a843bdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:44:01.319109 5881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82626ee153644a198ebd16ec6a843bdf [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:01.320267 5881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82626ee153644a198ebd16ec6a843bdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82626ee153644a198ebd16ec6a843bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41311 } }
I20260927 10:44:01.320652 5881 leader_election.cc:302] T 00000000000000000000000000000000 P 82626ee153644a198ebd16ec6a843bdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82626ee153644a198ebd16ec6a843bdf; no voters:
I20260927 10:44:01.321000 5881 leader_election.cc:288] T 00000000000000000000000000000000 P 82626ee153644a198ebd16ec6a843bdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:44:01.321175 5886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82626ee153644a198ebd16ec6a843bdf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:01.321445 5886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82626ee153644a198ebd16ec6a843bdf [term 1 LEADER]: Becoming Leader. State: Replica: 82626ee153644a198ebd16ec6a843bdf, State: Running, Role: LEADER
I20260927 10:44:01.321770 5886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82626ee153644a198ebd16ec6a843bdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82626ee153644a198ebd16ec6a843bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41311 } }
I20260927 10:44:01.322396 5881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82626ee153644a198ebd16ec6a843bdf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:44:01.323541 5888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82626ee153644a198ebd16ec6a843bdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82626ee153644a198ebd16ec6a843bdf. Latest consensus state: current_term: 1 leader_uuid: "82626ee153644a198ebd16ec6a843bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82626ee153644a198ebd16ec6a843bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41311 } } }
I20260927 10:44:01.323608 5887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82626ee153644a198ebd16ec6a843bdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82626ee153644a198ebd16ec6a843bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82626ee153644a198ebd16ec6a843bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41311 } } }
I20260927 10:44:01.323735 5887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82626ee153644a198ebd16ec6a843bdf [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:01.324144 5895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:44:01.323676 5888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82626ee153644a198ebd16ec6a843bdf [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:01.327513 5895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:44:01.333814 5895 catalog_manager.cc:1347] Generated new cluster ID: 9b941ea3fae44ab7b2d25591a0da7d4e
I20260927 10:44:01.333920 5895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:44:01.343086 5895 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:44:01.344357 5895 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:44:01.361953 5895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82626ee153644a198ebd16ec6a843bdf: Generated new TSK 0
I20260927 10:44:01.362934 5895 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:44:01.409024 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:01.379317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33022 (local address 127.0.98.62:41311)
0927 10:44:01.379520 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:01.379526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:01.379843 (+ 317us) server_negotiation.cc:407] Connection header received
0927 10:44:01.380027 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:01.380031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:01.380096 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:01.380231 (+ 135us) server_negotiation.cc:226] Negotiated authn=JWT
0927 10:44:01.380520 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:01.381351 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:01.382270 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:01.382647 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:01.382758 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:01.383319 (+ 561us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 10:44:01.408546 (+ 25227us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 10:44:01.408875 (+ 329us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33022: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":88}
W20260927 10:44:01.409107 5884 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33022: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 10:44:01.409479 5910 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:01.379209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41311 (local address 127.0.0.1:33022)
0927 10:44:01.379706 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:01.379731 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:01.379878 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:01.380283 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:01.380286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:01.380293 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0927 10:44:01.380403 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:01.380407 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:01.381489 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:01.381492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:01.382146 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:01.382154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:01.382841 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:01.382887 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:01.383015 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:01.383059 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 10:44:01.408800 (+ 25741us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 10:44:01.408998 (+ 198us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41311: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":254,"threads_started":1}
I20260927 10:44:01.410053 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 5814
2026-09-27T10:44:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (325 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (123 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (122 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:44:01 dist-test-slave-dzhw krb5kdc[5922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:44:01 dist-test-slave-dzhw krb5kdc[5922](info): set up 2 sockets
Sep 27 10:44:01 dist-test-slave-dzhw krb5kdc[5922](info): commencing operation
krb5kdc: starting...
W20260927 10:44:02.698866 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:44:02 dist-test-slave-dzhw krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:44:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:44:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:44:02.825266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36861
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:02.935832 5935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:02.936203 5935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:02.936249 5935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:02.940390 5935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:44:02.940464 5935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:02.940483 5935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:02.940500 5935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:44:02.940515 5935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:44:02.945971 5935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:02.947110 5935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:02.948115 5935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:44:02.957094 5935 server_base.cc:1034] running on GCE node
W20260927 10:44:02.957056 5940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:02.957053 5943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:02.957042 5941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:02.957863 5935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:02.959144 5935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:02.960338 5935 hybrid_clock.cc:648] HybridClock initialized: now 1790505842960328 us; error 26 us; skew 500 ppm
Sep 27 10:44:02 dist-test-slave-dzhw krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:02.967789 5935 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:44:02.968791 5935 webserver.cc:460] Webserver started at http://127.0.98.62:36709/ using document root <none> and password file <none>
I20260927 10:44:02.969326 5935 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:02.969386 5935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:02.969550 5935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:02.971632 5935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "82ec93fdb5514cd0b69c2d19af1e18e5"
format_stamp: "Formatted at 2026-09-27 10:44:02 on dist-test-slave-dzhw"
server_key: "a48783457d1dd24a20199b8079ec077d"
server_key_iv: "8e90745fc286b8ac091bd6486594f5b6"
server_key_version: "encryptionkey@0"
I20260927 10:44:02.972131 5935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "82ec93fdb5514cd0b69c2d19af1e18e5"
format_stamp: "Formatted at 2026-09-27 10:44:02 on dist-test-slave-dzhw"
server_key: "a48783457d1dd24a20199b8079ec077d"
server_key_iv: "8e90745fc286b8ac091bd6486594f5b6"
server_key_version: "encryptionkey@0"
I20260927 10:44:02.975915 5935 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:44:02.978309 5950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:02.979614 5935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:44:02.979753 5935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "82ec93fdb5514cd0b69c2d19af1e18e5"
format_stamp: "Formatted at 2026-09-27 10:44:02 on dist-test-slave-dzhw"
server_key: "a48783457d1dd24a20199b8079ec077d"
server_key_iv: "8e90745fc286b8ac091bd6486594f5b6"
server_key_version: "encryptionkey@0"
I20260927 10:44:02.979863 5935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:03.007380 5935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:03.010635 5935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:03.010856 5935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:03.017865 5935 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36861
I20260927 10:44:03.018180 6002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36861 every 8 connection(s)
I20260927 10:44:03.018924 5935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:44:03.021723 6003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:03.022163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 5935
I20260927 10:44:03.022321 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:44:03.022624 392 external_mini_cluster.cc:1428] Setting key 8eada96f5737f8600a33b1aa53c62d57
I20260927 10:44:03.028048 6003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82ec93fdb5514cd0b69c2d19af1e18e5: Bootstrap starting.
I20260927 10:44:03.030789 6003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82ec93fdb5514cd0b69c2d19af1e18e5: Neither blocks nor log segments found. Creating new log.
Sep 27 10:44:03 dist-test-slave-dzhw krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:44:03.031601 6003 log.cc:824] T 00000000000000000000000000000000 P 82ec93fdb5514cd0b69c2d19af1e18e5: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:03.033582 6003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82ec93fdb5514cd0b69c2d19af1e18e5: No bootstrap required, opened a new log
I20260927 10:44:03.036758 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:03.024094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38138 (local address 127.0.98.62:36861)
0927 10:44:03.024500 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:03.024516 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:03.024550 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:44:03.025171 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:03.025187 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:03.025519 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:03.025796 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:03.026736 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:03.027738 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:03.028830 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:03.029299 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:03.029421 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:03.032198 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:03.032204 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:03.032242 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:03.032270 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:03.034038 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:03.034501 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:03.034507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:03.034511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:03.034617 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:03.034908 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:03.034911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:03.034913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:03.035299 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:03.035493 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:03.035763 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 10:44:03.036036 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":138,"threads_started":1}
I20260927 10:44:03.037231 6003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82ec93fdb5514cd0b69c2d19af1e18e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ec93fdb5514cd0b69c2d19af1e18e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36861 } }
I20260927 10:44:03.037483 6003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82ec93fdb5514cd0b69c2d19af1e18e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:03.037570 6003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82ec93fdb5514cd0b69c2d19af1e18e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82ec93fdb5514cd0b69c2d19af1e18e5, State: Initialized, Role: FOLLOWER
I20260927 10:44:03.038061 6003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82ec93fdb5514cd0b69c2d19af1e18e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ec93fdb5514cd0b69c2d19af1e18e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36861 } }
I20260927 10:44:03.038215 6003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82ec93fdb5514cd0b69c2d19af1e18e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:44:03.038271 6003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82ec93fdb5514cd0b69c2d19af1e18e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:44:03.038374 6003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82ec93fdb5514cd0b69c2d19af1e18e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:03.039517 6003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82ec93fdb5514cd0b69c2d19af1e18e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ec93fdb5514cd0b69c2d19af1e18e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36861 } }
I20260927 10:44:03.039806 6003 leader_election.cc:302] T 00000000000000000000000000000000 P 82ec93fdb5514cd0b69c2d19af1e18e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82ec93fdb5514cd0b69c2d19af1e18e5; no voters:
I20260927 10:44:03.040066 6003 leader_election.cc:288] T 00000000000000000000000000000000 P 82ec93fdb5514cd0b69c2d19af1e18e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:44:03.040195 6008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82ec93fdb5514cd0b69c2d19af1e18e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:03.040438 6008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82ec93fdb5514cd0b69c2d19af1e18e5 [term 1 LEADER]: Becoming Leader. State: Replica: 82ec93fdb5514cd0b69c2d19af1e18e5, State: Running, Role: LEADER
I20260927 10:44:03.040729 6008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82ec93fdb5514cd0b69c2d19af1e18e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ec93fdb5514cd0b69c2d19af1e18e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36861 } }
I20260927 10:44:03.041291 6003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82ec93fdb5514cd0b69c2d19af1e18e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:44:03.042346 6009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82ec93fdb5514cd0b69c2d19af1e18e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82ec93fdb5514cd0b69c2d19af1e18e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ec93fdb5514cd0b69c2d19af1e18e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36861 } } }
I20260927 10:44:03.042521 6009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82ec93fdb5514cd0b69c2d19af1e18e5 [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:03.042790 6010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82ec93fdb5514cd0b69c2d19af1e18e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82ec93fdb5514cd0b69c2d19af1e18e5. Latest consensus state: current_term: 1 leader_uuid: "82ec93fdb5514cd0b69c2d19af1e18e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82ec93fdb5514cd0b69c2d19af1e18e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36861 } } }
I20260927 10:44:03.042889 6010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82ec93fdb5514cd0b69c2d19af1e18e5 [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:03.042975 6017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:44:03.046181 6017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:44:03.051873 6017 catalog_manager.cc:1347] Generated new cluster ID: bcfa0f64b4b74bcd8198470517b1a370
I20260927 10:44:03.052014 6017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:44:03.079556 6017 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:44:03.080909 6017 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:44:03.086639 6017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82ec93fdb5514cd0b69c2d19af1e18e5: Generated new TSK 0
I20260927 10:44:03.087455 6017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 10:44:03.117928 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36861
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:03.229460 6029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:03.229715 6029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:03.229758 6029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:03.233830 6029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:03.233886 6029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:03.233991 6029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:44:03.239109 6029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:03.240144 6029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:03.241148 6029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:03.250707 6034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:03.250747 6037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:03.250756 6035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:03.250818 6029 server_base.cc:1034] running on GCE node
I20260927 10:44:03.251394 6029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:03.252030 6029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:03.253226 6029 hybrid_clock.cc:648] HybridClock initialized: now 1790505843253191 us; error 45 us; skew 500 ppm
Sep 27 10:44:03 dist-test-slave-dzhw krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:03.255975 6029 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 10:44:03.256867 6029 webserver.cc:460] Webserver started at http://127.0.98.1:43169/ using document root <none> and password file <none>
I20260927 10:44:03.257427 6029 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:03.257488 6029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:03.257730 6029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:03.259857 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "3aab21344ec74c17b94ac8364278955c"
format_stamp: "Formatted at 2026-09-27 10:44:03 on dist-test-slave-dzhw"
server_key: "1e8ebefcb5f149e0f975c55aa4eece30"
server_key_iv: "bfd841cf5f9b1b8284c173c8cc0e0d37"
server_key_version: "encryptionkey@0"
I20260927 10:44:03.260439 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3aab21344ec74c17b94ac8364278955c"
format_stamp: "Formatted at 2026-09-27 10:44:03 on dist-test-slave-dzhw"
server_key: "1e8ebefcb5f149e0f975c55aa4eece30"
server_key_iv: "bfd841cf5f9b1b8284c173c8cc0e0d37"
server_key_version: "encryptionkey@0"
I20260927 10:44:03.264424 6029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:44:03.266827 6044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:03.268042 6029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:44:03.268173 6029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "3aab21344ec74c17b94ac8364278955c"
format_stamp: "Formatted at 2026-09-27 10:44:03 on dist-test-slave-dzhw"
server_key: "1e8ebefcb5f149e0f975c55aa4eece30"
server_key_iv: "bfd841cf5f9b1b8284c173c8cc0e0d37"
server_key_version: "encryptionkey@0"
I20260927 10:44:03.268283 6029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:03.297611 6029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:03.301651 6029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:03.301883 6029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:03.302544 6029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:03.303366 6029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:03.303411 6029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:03.303450 6029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:03.303465 6029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:03.312564 6029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35533
I20260927 10:44:03.312585 6157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35533 every 8 connection(s)
I20260927 10:44:03.313467 6029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:44:03.314334 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 6029
I20260927 10:44:03.314503 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:44:03.314771 392 external_mini_cluster.cc:1428] Setting key 34a494d69fdb63cad35fef708ec4e41a
Sep 27 10:44:03 dist-test-slave-dzhw krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 10:44:03.328408 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:03.315469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36875 (local address 127.0.98.62:36861)
0927 10:44:03.315603 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:03.315607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:03.316583 (+ 976us) server_negotiation.cc:407] Connection header received
0927 10:44:03.318150 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:03.318154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:03.318202 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:03.318273 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:03.319604 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:03.320148 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:03.321367 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:03.321600 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:03.321714 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:03.324280 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:03.324290 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:03.324293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:03.324315 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:03.325834 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:03.326362 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:03.326367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:03.326369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:03.326501 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:03.326952 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:03.326955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:03.326956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:03.327192 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:03.327320 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:03.328013 (+ 693us) server_negotiation.cc:299] Negotiation successful
0927 10:44:03.328159 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:44:03.329200 6160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:03.315932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36861 (local address 127.0.98.1:36875)
0927 10:44:03.316422 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:03.316474 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:03.317912 (+ 1438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:03.318484 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:03.318495 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:03.318978 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:03.319454 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:03.319466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:03.320325 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:03.320330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:03.321238 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:03.321247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:03.321775 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:03.321779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:03.321939 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:03.322609 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:03.322626 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:03.324098 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:03.325973 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:03.325979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:03.325995 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:03.326256 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:03.326650 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:03.326655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:03.326657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:03.326830 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:03.327314 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:03.327319 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:03.327695 (+ 376us) client_negotiation.cc:769] Sending connection context
0927 10:44:03.327920 (+ 225us) client_negotiation.cc:240] Negotiation successful
0927 10:44:03.328194 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":147,"threads_started":1}
I20260927 10:44:03.330639 6158 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36861
I20260927 10:44:03.330904 6158 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:03.331401 6158 heartbeater.cc:511] Master 127.0.98.62:36861 requested a full tablet report, sending...
I20260927 10:44:03.333158 5967 ts_manager.cc:194] Registered new tserver with Master: 3aab21344ec74c17b94ac8364278955c (127.0.98.1:35533)
I20260927 10:44:03.334833 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 10:44:03.346082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36861
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:03.455856 6163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:03.456130 6163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:03.456173 6163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:03.460285 6163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:03.460361 6163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:03.460461 6163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:44:03.465817 6163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:03.466949 6163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:03.467994 6163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:03.478041 6169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:03.478086 6163 server_base.cc:1034] running on GCE node
W20260927 10:44:03.478001 6171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:03.478001 6168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:03.478768 6163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:03.479462 6163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:03.480644 6163 hybrid_clock.cc:648] HybridClock initialized: now 1790505843480624 us; error 29 us; skew 500 ppm
Sep 27 10:44:03 dist-test-slave-dzhw krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:03.483587 6163 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 10:44:03.484462 6163 webserver.cc:460] Webserver started at http://127.0.98.2:45271/ using document root <none> and password file <none>
I20260927 10:44:03.484974 6163 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:03.485033 6163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:03.485201 6163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:03.487246 6163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "95c455779ffd4cd492edfb62b6eb938d"
format_stamp: "Formatted at 2026-09-27 10:44:03 on dist-test-slave-dzhw"
server_key: "61b15ac92628fb197f526296b25a3bab"
server_key_iv: "227bec8fc2d914032110be5c3c26af5c"
server_key_version: "encryptionkey@0"
I20260927 10:44:03.487742 6163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "95c455779ffd4cd492edfb62b6eb938d"
format_stamp: "Formatted at 2026-09-27 10:44:03 on dist-test-slave-dzhw"
server_key: "61b15ac92628fb197f526296b25a3bab"
server_key_iv: "227bec8fc2d914032110be5c3c26af5c"
server_key_version: "encryptionkey@0"
I20260927 10:44:03.491415 6163 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:44:03.493880 6178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:03.495205 6163 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:44:03.495347 6163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "95c455779ffd4cd492edfb62b6eb938d"
format_stamp: "Formatted at 2026-09-27 10:44:03 on dist-test-slave-dzhw"
server_key: "61b15ac92628fb197f526296b25a3bab"
server_key_iv: "227bec8fc2d914032110be5c3c26af5c"
server_key_version: "encryptionkey@0"
I20260927 10:44:03.495462 6163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:03.526199 6163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:03.529516 6163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:03.529742 6163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:03.530345 6163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:03.531212 6163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:03.531283 6163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:03.531358 6163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:03.531412 6163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:03.541501 6163 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37243
I20260927 10:44:03.541575 6291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37243 every 8 connection(s)
I20260927 10:44:03.542562 6163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 10:44:03 dist-test-slave-dzhw krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:44:03.552770 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 6163
I20260927 10:44:03.552893 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:44:03.553167 392 external_mini_cluster.cc:1428] Setting key 4b9b70e30c02d133557848bc98701181
I20260927 10:44:03.556499 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:03.544437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33891 (local address 127.0.98.62:36861)
0927 10:44:03.544566 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:03.544570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:03.545410 (+ 840us) server_negotiation.cc:407] Connection header received
0927 10:44:03.546878 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:03.546881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:03.546927 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:03.547003 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:03.548137 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:03.548666 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:03.549815 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:03.550042 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:03.550140 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:03.552535 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:03.552538 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:03.552541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:03.552563 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:03.554133 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:03.554783 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:03.554788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:03.554790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:03.554879 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:03.555241 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:03.555247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:03.555251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:03.555507 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:03.555611 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:03.556141 (+ 530us) server_negotiation.cc:299] Negotiation successful
0927 10:44:03.556269 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:44:03.557684 6294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:03.544822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36861 (local address 127.0.98.2:33891)
0927 10:44:03.545259 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:03.545302 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:03.546658 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:03.547191 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:03.547197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:03.547551 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:03.548011 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:03.548023 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:03.548794 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:03.548797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:03.549682 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:03.549690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:03.550207 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:03.550210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:03.550345 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:03.550942 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:03.550958 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:03.552380 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:03.554249 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:03.554255 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:03.554271 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:03.554687 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:03.554995 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:03.554998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:03.554999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:03.555132 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:03.555614 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:03.555619 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:03.555883 (+ 264us) client_negotiation.cc:769] Sending connection context
0927 10:44:03.556046 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 10:44:03.556519 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":112,"threads_started":1}
I20260927 10:44:03.559105 6292 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36861
I20260927 10:44:03.559393 6292 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:03.559891 6292 heartbeater.cc:511] Master 127.0.98.62:36861 requested a full tablet report, sending...
I20260927 10:44:03.560966 5967 ts_manager.cc:194] Registered new tserver with Master: 95c455779ffd4cd492edfb62b6eb938d (127.0.98.2:37243)
I20260927 10:44:03.561543 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33891
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 10:44:03.582755 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36861
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:03.691552 6297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:03.691816 6297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:03.691910 6297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:03.695982 6297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:03.696053 6297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:03.696151 6297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:44:03.701578 6297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:03.702695 6297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:03.703686 6297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:03.713666 6302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:03.713675 6305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:03.713738 6303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:03.713840 6297 server_base.cc:1034] running on GCE node
I20260927 10:44:03.714488 6297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:03.715099 6297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:03.716262 6297 hybrid_clock.cc:648] HybridClock initialized: now 1790505843716238 us; error 42 us; skew 500 ppm
Sep 27 10:44:03 dist-test-slave-dzhw krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:03.719122 6297 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 10:44:03.719920 6297 webserver.cc:460] Webserver started at http://127.0.98.3:38817/ using document root <none> and password file <none>
I20260927 10:44:03.720438 6297 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:03.720491 6297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:03.720649 6297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:03.722651 6297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "d56c307a3c4740c2863045bf70c8af57"
format_stamp: "Formatted at 2026-09-27 10:44:03 on dist-test-slave-dzhw"
server_key: "bfbaf2751a696bc4d928c5ec3481e17a"
server_key_iv: "bb9c2ff4133a3ab81b7827d7f1298e4e"
server_key_version: "encryptionkey@0"
I20260927 10:44:03.723140 6297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d56c307a3c4740c2863045bf70c8af57"
format_stamp: "Formatted at 2026-09-27 10:44:03 on dist-test-slave-dzhw"
server_key: "bfbaf2751a696bc4d928c5ec3481e17a"
server_key_iv: "bb9c2ff4133a3ab81b7827d7f1298e4e"
server_key_version: "encryptionkey@0"
I20260927 10:44:03.726708 6297 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:44:03.729277 6312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:03.730625 6297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:44:03.730772 6297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "d56c307a3c4740c2863045bf70c8af57"
format_stamp: "Formatted at 2026-09-27 10:44:03 on dist-test-slave-dzhw"
server_key: "bfbaf2751a696bc4d928c5ec3481e17a"
server_key_iv: "bb9c2ff4133a3ab81b7827d7f1298e4e"
server_key_version: "encryptionkey@0"
I20260927 10:44:03.730890 6297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:03.751375 6297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:03.754801 6297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:03.755012 6297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:03.755592 6297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:03.756423 6297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:03.756494 6297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:03.756564 6297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:03.756616 6297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:03.765407 6297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34319
I20260927 10:44:03.765453 6425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34319 every 8 connection(s)
I20260927 10:44:03.766350 6297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:44:03.769253 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 6297
I20260927 10:44:03.769380 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:44:03.769696 392 external_mini_cluster.cc:1428] Setting key 9590d85f304341eef302efc61eabcb50
Sep 27 10:44:03 dist-test-slave-dzhw krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:44:03.781178 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:03.768301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48353 (local address 127.0.98.62:36861)
0927 10:44:03.768441 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:03.768445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:03.769439 (+ 994us) server_negotiation.cc:407] Connection header received
0927 10:44:03.771316 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:03.771319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:03.771375 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:03.771464 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:03.772815 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:03.773460 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:03.774696 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:03.774933 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:03.775023 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:03.777667 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:03.777670 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:03.777673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:03.777706 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:03.779144 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:03.779677 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:03.779682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:03.779684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:03.779764 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:03.780100 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:03.780105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:03.780106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:03.780295 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:03.780401 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:03.780897 (+ 496us) server_negotiation.cc:299] Negotiation successful
0927 10:44:03.781014 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:44:03.781989 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:03.768676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36861 (local address 127.0.98.3:48353)
0927 10:44:03.769243 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:03.769328 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:03.770995 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:03.771622 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:03.771629 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:03.772131 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:03.772683 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:03.772697 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:03.773566 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:03.773568 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:03.774582 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:03.774592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:03.775093 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:03.775095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:03.775242 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:03.775938 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:03.775954 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:03.777493 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:03.779278 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:03.779284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:03.779300 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:03.779573 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:03.779859 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:03.779862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:03.779864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:03.780005 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:03.780398 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:03.780404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:03.780659 (+ 255us) client_negotiation.cc:769] Sending connection context
0927 10:44:03.780859 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 10:44:03.781175 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":125,"threads_started":1}
I20260927 10:44:03.783310 6426 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36861
I20260927 10:44:03.783566 6426 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:03.784152 6426 heartbeater.cc:511] Master 127.0.98.62:36861 requested a full tablet report, sending...
I20260927 10:44:03.785048 5967 ts_manager.cc:194] Registered new tserver with Master: d56c307a3c4740c2863045bf70c8af57 (127.0.98.3:34319)
I20260927 10:44:03.785579 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48353
I20260927 10:44:03.795892 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/test-admin.keytab.
Sep 27 10:44:03 dist-test-slave-dzhw krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:03.810768 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 10:44:03.813542 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:03.812102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59046 (local address 127.0.98.1:35533)
0927 10:44:03.812453 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:03.812476 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:03.812497 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:44:03.812544 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:03.812549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:03.812734 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:03.812887 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:03.813356 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59046: BlockingRecv error: recv got EOF from 127.0.0.1:59046 (error 108)
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":112,"threads_started":1}
W20260927 10:44:03.813807 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/krb5cc)
Sep 27 10:44:03 dist-test-slave-dzhw krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:03.814914 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 10:44:03 dist-test-slave-dzhw krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:03.815824 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 10:44:03 dist-test-slave-dzhw krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 10:44:03.825590 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:03.816682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59054 (local address 127.0.98.1:35533)
0927 10:44:03.816809 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:03.816813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:03.816911 (+ 98us) server_negotiation.cc:407] Connection header received
0927 10:44:03.817059 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:03.817062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:03.817109 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:03.817194 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:03.817705 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:03.818537 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:03.819356 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:03.819728 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:03.819841 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:03.822166 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:03.822176 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:03.822201 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:03.822223 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:03.823864 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:03.824308 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:03.824314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:03.824315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:03.824391 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:03.824659 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:03.824664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:03.824665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:03.824900 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:03.825104 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:03.825288 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 10:44:03.825443 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 10:44:03.829308 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:03.828403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59058 (local address 127.0.98.1:35533)
0927 10:44:03.828525 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:03.828529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:03.828669 (+ 140us) server_negotiation.cc:407] Connection header received
0927 10:44:03.828756 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:03.828758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:03.828792 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:03.828852 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:03.829220 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59058: BlockingRecv error: recv got EOF from 127.0.0.1:59058 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 10:44:03.829439 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 10:44:03 dist-test-slave-dzhw krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:03.830499 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 10:44:03 dist-test-slave-dzhw krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:03.831341 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 10:44:03 dist-test-slave-dzhw krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 10:44:03.839615 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:03.832075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59064 (local address 127.0.98.1:35533)
0927 10:44:03.832186 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:03.832190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:03.832282 (+ 92us) server_negotiation.cc:407] Connection header received
0927 10:44:03.832365 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:03.832368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:03.832399 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:03.832465 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:03.832902 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:03.833396 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:03.834147 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:03.834438 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:03.834553 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:03.836754 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:03.836757 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:03.836759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:03.836781 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:03.838160 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:03.838589 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:03.838591 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:03.838592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:03.838659 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:03.838920 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:03.838922 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:03.838923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:03.839061 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:03.839148 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:03.839412 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 10:44:03.839501 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 10:44:03.842368 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:03.841322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59076 (local address 127.0.98.1:35533)
0927 10:44:03.841446 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:03.841451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:03.841515 (+ 64us) server_negotiation.cc:407] Connection header received
0927 10:44:03.841642 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:03.841645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:03.841686 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:03.841771 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:03.842253 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59076: BlockingRecv error: recv got EOF from 127.0.0.1:59076 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 10:44:03.842487 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505812134190-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 10:44:03 dist-test-slave-dzhw krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:03.843588 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 10:44:03 dist-test-slave-dzhw krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:03.844486 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 10:44:03 dist-test-slave-dzhw krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 10:44:03.852986 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:03.845303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59090 (local address 127.0.98.1:35533)
0927 10:44:03.845417 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:03.845421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:03.845517 (+ 96us) server_negotiation.cc:407] Connection header received
0927 10:44:03.845659 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:03.845662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:03.845710 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:03.845794 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:03.846283 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:03.846853 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:03.847636 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:03.847863 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:03.847971 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:03.850029 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:03.850033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:03.850035 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:03.850067 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:03.851353 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:03.851799 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:03.851805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:03.851807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:03.851882 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:03.852152 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:03.852157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:03.852158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:03.852356 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:03.852495 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:03.852723 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 10:44:03.852827 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:44:03.854184 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 6029
I20260927 10:44:03.860054 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 6163
I20260927 10:44:03.865168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 6297
I20260927 10:44:03.870288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 5935
2026-09-27T10:44:03Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2217 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:44:03 dist-test-slave-dzhw krb5kdc[6446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:44:03 dist-test-slave-dzhw krb5kdc[6446](info): set up 2 sockets
Sep 27 10:44:03 dist-test-slave-dzhw krb5kdc[6446](info): commencing operation
krb5kdc: starting...
W20260927 10:44:04.906884 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:44:04 dist-test-slave-dzhw krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:44:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:44:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 10:44:05.027676 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34339
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:05.138988 6459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:05.139299 6459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:05.139391 6459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:05.143527 6459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:44:05.143620 6459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:05.143657 6459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:05.143821 6459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:44:05.143922 6459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:44:05.149456 6459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:05.150686 6459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:05.151736 6459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:05.160487 6467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:05.160552 6464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:05.160701 6459 server_base.cc:1034] running on GCE node
W20260927 10:44:05.160554 6465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:05.161311 6459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:05.162508 6459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:05.163718 6459 hybrid_clock.cc:648] HybridClock initialized: now 1790505845163689 us; error 44 us; skew 500 ppm
Sep 27 10:44:05 dist-test-slave-dzhw krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:05.166638 6459 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 10:44:05.167506 6459 webserver.cc:460] Webserver started at http://127.0.98.62:39601/ using document root <none> and password file <none>
I20260927 10:44:05.168095 6459 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:05.168157 6459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:05.168326 6459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:05.170372 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "ed2c254b7be546a19bad6932d621df69"
format_stamp: "Formatted at 2026-09-27 10:44:05 on dist-test-slave-dzhw"
server_key: "12021f5aacd0b764f231c33ef002c5a7"
server_key_iv: "24b664206da0990c95e2394d3e91c098"
server_key_version: "encryptionkey@0"
I20260927 10:44:05.170976 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "ed2c254b7be546a19bad6932d621df69"
format_stamp: "Formatted at 2026-09-27 10:44:05 on dist-test-slave-dzhw"
server_key: "12021f5aacd0b764f231c33ef002c5a7"
server_key_iv: "24b664206da0990c95e2394d3e91c098"
server_key_version: "encryptionkey@0"
I20260927 10:44:05.174723 6459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 10:44:05.177177 6474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:05.178594 6459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:44:05.178732 6459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "ed2c254b7be546a19bad6932d621df69"
format_stamp: "Formatted at 2026-09-27 10:44:05 on dist-test-slave-dzhw"
server_key: "12021f5aacd0b764f231c33ef002c5a7"
server_key_iv: "24b664206da0990c95e2394d3e91c098"
server_key_version: "encryptionkey@0"
I20260927 10:44:05.178844 6459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:05.192394 6459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:05.195873 6459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:05.196084 6459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:05.202996 6459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34339
I20260927 10:44:05.203320 6526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34339 every 8 connection(s)
I20260927 10:44:05.204088 6459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:44:05.207041 6527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:05.213325 6527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69: Bootstrap starting.
I20260927 10:44:05.214130 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 6459
I20260927 10:44:05.214313 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:44:05.214638 392 external_mini_cluster.cc:1428] Setting key 3828357086fa9d4ed81be914da28ef8d
I20260927 10:44:05.216117 6527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:05.217005 6527 log.cc:824] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:05.219198 6527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69: No bootstrap required, opened a new log
I20260927 10:44:05.223300 6527 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } }
I20260927 10:44:05.223553 6527 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:05.223637 6527 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2c254b7be546a19bad6932d621df69, State: Initialized, Role: FOLLOWER
Sep 27 10:44:05 dist-test-slave-dzhw krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 10:44:05.224166 6527 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } }
I20260927 10:44:05.224328 6527 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:44:05.224397 6527 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:44:05.224493 6527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:05.225415 6527 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } }
I20260927 10:44:05.225812 6527 leader_election.cc:302] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed2c254b7be546a19bad6932d621df69; no voters:
I20260927 10:44:05.226186 6527 leader_election.cc:288] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:44:05.226285 6532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:05.226542 6532 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [term 1 LEADER]: Becoming Leader. State: Replica: ed2c254b7be546a19bad6932d621df69, State: Running, Role: LEADER
I20260927 10:44:05.226853 6532 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } }
I20260927 10:44:05.227602 6527 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:44:05.228144 6532 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed2c254b7be546a19bad6932d621df69. Latest consensus state: current_term: 1 leader_uuid: "ed2c254b7be546a19bad6932d621df69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } }
I20260927 10:44:05.228938 6532 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:05.228835 6534 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed2c254b7be546a19bad6932d621df69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } }
I20260927 10:44:05.229238 6534 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:05.230831 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:05.215832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36624 (local address 127.0.98.62:34339)
0927 10:44:05.216327 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:05.216349 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:05.216397 (+ 48us) server_negotiation.cc:407] Connection header received
0927 10:44:05.217257 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:05.217294 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:05.217648 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:05.218026 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:05.218727 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:05.220173 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:05.221097 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:05.221829 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:05.222032 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:05.224734 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:05.224739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:05.224752 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:05.224780 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:05.226830 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:05.227402 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:05.227408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:05.227414 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:05.227550 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:05.227850 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:05.227853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:05.227855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:05.228397 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:05.228618 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:05.229521 (+ 903us) server_negotiation.cc:299] Negotiation successful
0927 10:44:05.229833 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":123,"threads_started":1}
I20260927 10:44:05.230189 6541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:44:05.234030 6541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:44:05.239650 6541 catalog_manager.cc:1347] Generated new cluster ID: 8d324f4216d949b7b8161f188793fea6
I20260927 10:44:05.239727 6541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:44:05.249912 6541 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:44:05.250905 6541 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:44:05.263845 6541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69: Generated new TSK 0
I20260927 10:44:05.264726 6541 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 10:44:05.302642 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34339
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:05.414873 6553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:05.415202 6553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:05.415285 6553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:05.419314 6553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:05.419414 6553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:05.419508 6553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:44:05.424769 6553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:05.425885 6553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:05.426947 6553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:44:05.437119 6553 server_base.cc:1034] running on GCE node
W20260927 10:44:05.437063 6558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:05.437063 6561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:05.437073 6559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:05.437806 6553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:05.438583 6553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:05.439783 6553 hybrid_clock.cc:648] HybridClock initialized: now 1790505845439771 us; error 37 us; skew 500 ppm
Sep 27 10:44:05 dist-test-slave-dzhw krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:05.442623 6553 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 10:44:05.443454 6553 webserver.cc:460] Webserver started at http://127.0.98.1:46001/ using document root <none> and password file <none>
I20260927 10:44:05.443965 6553 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:05.444022 6553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:05.444183 6553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:05.446130 6553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "e5c1e1840c784fdd871dc6b3572c1e83"
format_stamp: "Formatted at 2026-09-27 10:44:05 on dist-test-slave-dzhw"
server_key: "d208e4876c4cdf3873875961de6bfaf3"
server_key_iv: "8ade8502c08046333b8e2ac710be661d"
server_key_version: "encryptionkey@0"
I20260927 10:44:05.446705 6553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e5c1e1840c784fdd871dc6b3572c1e83"
format_stamp: "Formatted at 2026-09-27 10:44:05 on dist-test-slave-dzhw"
server_key: "d208e4876c4cdf3873875961de6bfaf3"
server_key_iv: "8ade8502c08046333b8e2ac710be661d"
server_key_version: "encryptionkey@0"
I20260927 10:44:05.450366 6553 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 10:44:05.452870 6568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:05.454129 6553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:44:05.454279 6553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "e5c1e1840c784fdd871dc6b3572c1e83"
format_stamp: "Formatted at 2026-09-27 10:44:05 on dist-test-slave-dzhw"
server_key: "d208e4876c4cdf3873875961de6bfaf3"
server_key_iv: "8ade8502c08046333b8e2ac710be661d"
server_key_version: "encryptionkey@0"
I20260927 10:44:05.454394 6553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:05.467619 6553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:05.471094 6553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:05.471305 6553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:05.471904 6553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:05.472793 6553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:05.472841 6553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:05.472908 6553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:05.472937 6553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:05.482311 6553 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38249
I20260927 10:44:05.482373 6681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38249 every 8 connection(s)
I20260927 10:44:05.483352 6553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:44:05.489440 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 6553
I20260927 10:44:05.489583 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:44:05.489857 392 external_mini_cluster.cc:1428] Setting key f822cead4666f51259ad734bf441d0d9
Sep 27 10:44:05 dist-test-slave-dzhw krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 10:44:05.498245 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:05.485261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36773 (local address 127.0.98.62:34339)
0927 10:44:05.485410 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:05.485414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:05.486272 (+ 858us) server_negotiation.cc:407] Connection header received
0927 10:44:05.487651 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:05.487654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:05.487702 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:05.487781 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:05.488981 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:05.489551 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:05.491018 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:05.491317 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:05.491419 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:05.494215 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:05.494218 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:05.494220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:05.494244 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:05.495818 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:05.496494 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:05.496497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:05.496498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:05.496575 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:05.497028 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:05.497031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:05.497032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:05.497223 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:05.497347 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:05.498018 (+ 671us) server_negotiation.cc:299] Negotiation successful
0927 10:44:05.498121 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:44:05.499564 6684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:05.485641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.1:36773)
0927 10:44:05.486093 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:05.486144 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:05.487455 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:05.487948 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:05.487955 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:05.488335 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:05.488820 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:05.488832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:05.489756 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:05.489759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:05.490886 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:05.490896 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:05.491502 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:05.491505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:05.491654 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:05.492289 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:05.492351 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:05.493993 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:05.495948 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:05.495957 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:05.495976 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:05.496378 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:05.496675 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:05.496681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:05.496685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:05.496899 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:05.497357 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:05.497366 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:05.497742 (+ 376us) client_negotiation.cc:769] Sending connection context
0927 10:44:05.497987 (+ 245us) client_negotiation.cc:240] Negotiation successful
0927 10:44:05.498375 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 10:44:05.500852 6682 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34339
I20260927 10:44:05.501118 6682 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:05.501590 6682 heartbeater.cc:511] Master 127.0.98.62:34339 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260927 10:44:05.503214 6491 ts_manager.cc:194] Registered new tserver with Master: e5c1e1840c784fdd871dc6b3572c1e83 (127.0.98.1:38249)
I20260927 10:44:05.504729 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 10:44:05.522229 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34339
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:05.633252 6687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:05.633584 6687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:05.633633 6687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:05.637733 6687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:05.637792 6687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:05.637856 6687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:44:05.645260 6687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:05.646778 6687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:05.648118 6687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:05.658378 6695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:05.658396 6692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:05.658488 6687 server_base.cc:1034] running on GCE node
W20260927 10:44:05.658396 6693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:05.659047 6687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:05.659770 6687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:05.660959 6687 hybrid_clock.cc:648] HybridClock initialized: now 1790505845660945 us; error 27 us; skew 500 ppm
Sep 27 10:44:05 dist-test-slave-dzhw krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:05.663820 6687 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 10:44:05.664690 6687 webserver.cc:460] Webserver started at http://127.0.98.2:38753/ using document root <none> and password file <none>
I20260927 10:44:05.665212 6687 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:05.665279 6687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:05.665445 6687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:05.667534 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "ecaafcecaf3e414984d0c826a0c4246d"
format_stamp: "Formatted at 2026-09-27 10:44:05 on dist-test-slave-dzhw"
server_key: "fd08cb6245077e8fdcd5508114e172d7"
server_key_iv: "4a859f10d455fc8ffe5d3fcdced2a747"
server_key_version: "encryptionkey@0"
I20260927 10:44:05.668107 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ecaafcecaf3e414984d0c826a0c4246d"
format_stamp: "Formatted at 2026-09-27 10:44:05 on dist-test-slave-dzhw"
server_key: "fd08cb6245077e8fdcd5508114e172d7"
server_key_iv: "4a859f10d455fc8ffe5d3fcdced2a747"
server_key_version: "encryptionkey@0"
I20260927 10:44:05.671772 6687 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:44:05.674171 6702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:05.675462 6687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:44:05.675562 6687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "ecaafcecaf3e414984d0c826a0c4246d"
format_stamp: "Formatted at 2026-09-27 10:44:05 on dist-test-slave-dzhw"
server_key: "fd08cb6245077e8fdcd5508114e172d7"
server_key_iv: "4a859f10d455fc8ffe5d3fcdced2a747"
server_key_version: "encryptionkey@0"
I20260927 10:44:05.675650 6687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:05.705876 6687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:05.709306 6687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:05.709461 6687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:05.709982 6687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:05.710772 6687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:05.710817 6687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:05.710856 6687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:05.710871 6687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:05.719882 6687 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44599
I20260927 10:44:05.719996 6815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44599 every 8 connection(s)
I20260927 10:44:05.720831 6687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:44:05.728971 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 6687
I20260927 10:44:05.729094 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:44:05.729492 392 external_mini_cluster.cc:1428] Setting key d722e1486f2d54a5f6ff7aab3ecb58fd
Sep 27 10:44:05 dist-test-slave-dzhw krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 10:44:05.735991 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:05.722704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37711 (local address 127.0.98.62:34339)
0927 10:44:05.722843 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:05.722847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:05.723639 (+ 792us) server_negotiation.cc:407] Connection header received
0927 10:44:05.724931 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:05.724934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:05.724979 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:05.725061 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:05.726211 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:05.726832 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:05.728105 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:05.728372 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:05.728465 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:05.731935 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:05.731938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:05.731940 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:05.731962 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:05.733655 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:05.734161 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:05.734168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:05.734173 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:05.734262 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:05.734659 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:05.734665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:05.734669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:05.734924 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:05.735033 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:05.735597 (+ 564us) server_negotiation.cc:299] Negotiation successful
0927 10:44:05.735735 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:44:05.736680 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:05.723044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.2:37711)
0927 10:44:05.723478 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:05.723520 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:05.724719 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:05.725217 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:05.725223 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:05.725592 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:05.726053 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:05.726065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:05.726970 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:05.726973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:05.727990 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:05.728000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:05.728542 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:05.728545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:05.728740 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:05.729694 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:05.729738 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:05.731768 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:05.733786 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:05.733791 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:05.733803 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:05.734059 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:05.734356 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:05.734358 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:05.734360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:05.734558 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:05.735027 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:05.735032 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:05.735351 (+ 319us) client_negotiation.cc:769] Sending connection context
0927 10:44:05.735511 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 10:44:05.735767 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":113,"threads_started":1}
I20260927 10:44:05.737885 6816 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34339
I20260927 10:44:05.738155 6816 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:05.738794 6816 heartbeater.cc:511] Master 127.0.98.62:34339 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 10:44:05.740024 6491 ts_manager.cc:194] Registered new tserver with Master: ecaafcecaf3e414984d0c826a0c4246d (127.0.98.2:44599)
I20260927 10:44:05.740610 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 10:44:05.758644 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34339
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:05.869318 6821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:05.869619 6821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:05.869724 6821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:05.873736 6821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:05.873827 6821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:05.873921 6821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:44:05.879176 6821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:05.880298 6821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:05.881304 6821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:05.890897 6827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:05.890947 6826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:05.890879 6829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:05.891054 6821 server_base.cc:1034] running on GCE node
I20260927 10:44:05.891557 6821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:05.892132 6821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:05.893297 6821 hybrid_clock.cc:648] HybridClock initialized: now 1790505845893258 us; error 53 us; skew 500 ppm
Sep 27 10:44:05 dist-test-slave-dzhw krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:05.896087 6821 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 10:44:05.897032 6821 webserver.cc:460] Webserver started at http://127.0.98.3:43719/ using document root <none> and password file <none>
I20260927 10:44:05.897614 6821 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:05.897723 6821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:05.897956 6821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:05.900045 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "babeb35fec414bac989abcc216118c74"
format_stamp: "Formatted at 2026-09-27 10:44:05 on dist-test-slave-dzhw"
server_key: "d6166917264acbbee766c1148d1ea695"
server_key_iv: "cbfa7252c76fdb43ab3465c22da30b91"
server_key_version: "encryptionkey@0"
I20260927 10:44:05.900614 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "babeb35fec414bac989abcc216118c74"
format_stamp: "Formatted at 2026-09-27 10:44:05 on dist-test-slave-dzhw"
server_key: "d6166917264acbbee766c1148d1ea695"
server_key_iv: "cbfa7252c76fdb43ab3465c22da30b91"
server_key_version: "encryptionkey@0"
I20260927 10:44:05.904464 6821 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:44:05.906935 6836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:05.908181 6821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:44:05.908330 6821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "babeb35fec414bac989abcc216118c74"
format_stamp: "Formatted at 2026-09-27 10:44:05 on dist-test-slave-dzhw"
server_key: "d6166917264acbbee766c1148d1ea695"
server_key_iv: "cbfa7252c76fdb43ab3465c22da30b91"
server_key_version: "encryptionkey@0"
I20260927 10:44:05.908452 6821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:05.917546 6821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:05.920485 6821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:05.920689 6821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:05.921259 6821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:05.922076 6821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:05.922149 6821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:05.922215 6821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:05.922266 6821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:05.931625 6821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44633
I20260927 10:44:05.931646 6949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44633 every 8 connection(s)
I20260927 10:44:05.932510 6821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:44:05.934698 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 6821
I20260927 10:44:05.934824 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:44:05.935168 392 external_mini_cluster.cc:1428] Setting key fc3c433d0c60e194cd4ceb3ea7348cbf
Sep 27 10:44:05 dist-test-slave-dzhw krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 10:44:05.946144 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:05.934315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34225 (local address 127.0.98.62:34339)
0927 10:44:05.934538 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:05.934545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:05.935459 (+ 914us) server_negotiation.cc:407] Connection header received
0927 10:44:05.936939 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:05.936945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:05.937000 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:05.937082 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:05.938242 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:05.938836 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:05.939966 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:05.940195 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:05.940307 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:05.942754 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:05.942758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:05.942761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:05.942791 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:05.943992 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:05.944547 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:05.944552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:05.944554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:05.944635 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:05.945039 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:05.945044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:05.945046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:05.945240 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:05.945348 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:05.945863 (+ 515us) server_negotiation.cc:299] Negotiation successful
0927 10:44:05.945980 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 10:44:05.947017 6952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:05.934733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.3:34225)
0927 10:44:05.935293 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:05.935348 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:05.936753 (+ 1405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:05.937260 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:05.937269 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:05.937629 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:05.938097 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:05.938109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:05.938955 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:05.938958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:05.939831 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:05.939841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:05.940353 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:05.940358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:05.940531 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:05.941075 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:05.941091 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:05.942578 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:05.944117 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:05.944122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:05.944139 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:05.944439 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:05.944734 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:05.944737 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:05.944748 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:05.944939 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:05.945344 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:05.945350 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:05.945622 (+ 272us) client_negotiation.cc:769] Sending connection context
0927 10:44:05.945823 (+ 201us) client_negotiation.cc:240] Negotiation successful
0927 10:44:05.946140 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":122,"threads_started":1}
I20260927 10:44:05.948340 6950 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34339
I20260927 10:44:05.948598 6950 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:05.949133 6950 heartbeater.cc:511] Master 127.0.98.62:34339 requested a full tablet report, sending...
I20260927 10:44:05.950093 6491 ts_manager.cc:194] Registered new tserver with Master: babeb35fec414bac989abcc216118c74 (127.0.98.3:44633)
I20260927 10:44:05.950729 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34225
I20260927 10:44:05.960754 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 10:44:05.987710 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44553
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34339,127.0.98.61:44553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:06.097131 6955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:06.097470 6955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:06.097514 6955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:06.101542 6955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:44:06.101608 6955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:06.101629 6955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:06.101645 6955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:44:06.101661 6955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:44:06.107035 6955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34339,127.0.98.61:44553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
W20260927 10:44:06.107249 6955 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:34339,127.0.98.61:44553'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 10:44:06.108103 6955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:06.109046 6955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:06.118009 6963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:06.118028 6960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:06.118067 6961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:06.118088 6955 server_base.cc:1034] running on GCE node
I20260927 10:44:06.118856 6955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:06.119995 6955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:06.121188 6955 hybrid_clock.cc:648] HybridClock initialized: now 1790505846121162 us; error 52 us; skew 500 ppm
Sep 27 10:44:06 dist-test-slave-dzhw krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:06.124015 6955 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 10:44:06.124895 6955 webserver.cc:460] Webserver started at http://127.0.98.61:45571/ using document root <none> and password file <none>
I20260927 10:44:06.125449 6955 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:06.125540 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:06.125778 6955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:06.127965 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/data/instance:
uuid: "258937809ebb4a2098f5a0f9d45a9ecb"
format_stamp: "Formatted at 2026-09-27 10:44:06 on dist-test-slave-dzhw"
server_key: "d5f2148f36e74e2675f275474269fd3f"
server_key_iv: "253899e787cc5647d95e9277d7acb14a"
server_key_version: "encryptionkey@0"
I20260927 10:44:06.128592 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/wal/instance:
uuid: "258937809ebb4a2098f5a0f9d45a9ecb"
format_stamp: "Formatted at 2026-09-27 10:44:06 on dist-test-slave-dzhw"
server_key: "d5f2148f36e74e2675f275474269fd3f"
server_key_iv: "253899e787cc5647d95e9277d7acb14a"
server_key_version: "encryptionkey@0"
I20260927 10:44:06.132483 6955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:44:06.135037 6970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:06.136320 6955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:44:06.136485 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/wal
uuid: "258937809ebb4a2098f5a0f9d45a9ecb"
format_stamp: "Formatted at 2026-09-27 10:44:06 on dist-test-slave-dzhw"
server_key: "d5f2148f36e74e2675f275474269fd3f"
server_key_iv: "253899e787cc5647d95e9277d7acb14a"
server_key_version: "encryptionkey@0"
I20260927 10:44:06.136606 6955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:06.158977 6955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:06.162267 6955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:06.162513 6955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:06.169612 6955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44553
I20260927 10:44:06.169684 7022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44553 every 8 connection(s)
I20260927 10:44:06.170598 6955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/data/info.pb
I20260927 10:44:06.173326 7023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:06.173884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 6955
I20260927 10:44:06.177937 7023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:44:06 dist-test-slave-dzhw krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 10:44:06 dist-test-slave-dzhw krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 10:44:06.192965 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.181625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36626 (local address 127.0.98.62:34339)
0927 10:44:06.181765 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:06.181771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:06.181786 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:44:06.181832 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:06.181837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:06.181892 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:06.181974 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:06.183045 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.183816 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.185154 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.185475 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.185556 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.188656 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:06.188659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:06.188662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:06.188696 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:06.190665 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.191305 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.191311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.191316 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.191408 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.191800 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.191806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.191808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.192134 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:06.192254 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:06.192683 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 10:44:06.192802 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:44:06.194556 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.180055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.0.1:36626)
0927 10:44:06.181259 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:06.181313 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:06.181430 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:06.182154 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:06.182162 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:06.182664 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:06.182903 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.182911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.183981 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.183984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.185001 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.185011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.185766 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.185773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.185878 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.186503 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:06.186545 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:06.188524 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:06.190790 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.190798 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.190809 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.191151 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.191499 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.191501 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.191502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.191658 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.192258 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:06.192267 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:06.192439 (+ 172us) client_negotiation.cc:769] Sending connection context
0927 10:44:06.193114 (+ 675us) client_negotiation.cc:240] Negotiation successful
0927 10:44:06.193425 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"mutex_wait_us":59,"thread_start_us":153,"threads_started":1}
I20260927 10:44:06.194557 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.175363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52490 (local address 127.0.98.61:44553)
0927 10:44:06.175775 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:06.175789 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:06.175823 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:44:06.176457 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:06.176473 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:06.176776 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:06.177102 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:06.177920 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.179873 (+ 1953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.181508 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.182729 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.183654 (+ 925us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.187676 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:06.187684 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:06.187698 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:06.187729 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:06.190359 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.190965 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.190973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.190977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.191162 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.191637 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.191643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.191648 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.192244 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:06.192476 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:06.192720 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 10:44:06.193344 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"spinlock_wait_cycles":1792,"thread_start_us":135,"threads_started":1}
I20260927 10:44:06.196046 7023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:44:06 dist-test-slave-dzhw krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 10:44:06.205869 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.196543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44553 (local address 127.0.0.1:52500)
0927 10:44:06.196776 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:06.196789 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:06.196863 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:06.197104 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:06.197108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:06.197293 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:06.197407 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.197413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.198827 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.198831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.199693 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.199706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.200274 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.200278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.200379 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.200920 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:06.200931 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:06.202252 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:06.204343 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.204346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.204349 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.204577 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.204873 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.204875 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.204877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.204953 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.205379 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:06.205382 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:06.205458 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 10:44:06.205548 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 10:44:06.205683 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260927 10:44:06.206501 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.196772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52500 (local address 127.0.98.61:44553)
0927 10:44:06.196893 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:06.196900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:06.196913 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:44:06.196947 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:06.196950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:06.196989 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:06.197069 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:06.197497 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.198699 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.199832 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.200134 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.200240 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.202409 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:06.202435 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:06.202438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:06.202461 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:06.204223 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.204676 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.204680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.204682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.204755 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.205042 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.205045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.205047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.205255 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:06.205345 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:06.206087 (+ 742us) server_negotiation.cc:299] Negotiation successful
0927 10:44:06.206206 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:44:06.209683 7023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb: Bootstrap starting.
I20260927 10:44:06.212373 7023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:06.213215 7023 log.cc:824] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:06.215289 7023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb: No bootstrap required, opened a new log
I20260927 10:44:06.219025 7023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } }
I20260927 10:44:06.219218 7023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:06.219259 7023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 258937809ebb4a2098f5a0f9d45a9ecb, State: Initialized, Role: FOLLOWER
I20260927 10:44:06.219686 7023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } }
I20260927 10:44:06.220103 7029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } } }
I20260927 10:44:06.220237 7029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:44:06.220863 7023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [sys.catalog]: configured and running, proceeding with master startup.
W20260927 10:44:06.225589 7040 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:44:06.225682 7040 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:44:06.227867 6955 master_runner.cc:428] Detected that this master 258937809ebb4a2098f5a0f9d45a9ecb is joining an existing cluster
I20260927 10:44:06.227967 6955 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:44553
I20260927 10:44:06.239302 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.230188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36638 (local address 127.0.98.62:34339)
0927 10:44:06.230328 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:06.230334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:06.230591 (+ 257us) server_negotiation.cc:407] Connection header received
0927 10:44:06.230788 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:06.230791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:06.230842 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:06.230900 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:06.231468 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.232274 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.233111 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.233449 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.233552 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.235718 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:06.235722 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:06.235725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:06.235752 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:06.237464 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.237947 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.237950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.237952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.238062 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.238339 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.238342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.238344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.238600 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:06.238688 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:06.239002 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 10:44:06.239124 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:44:06.239310 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.230689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52506 (local address 127.0.98.61:44553)
0927 10:44:06.230804 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:06.230808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:06.231696 (+ 888us) server_negotiation.cc:407] Connection header received
0927 10:44:06.231783 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:06.231786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:06.231831 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:06.231912 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:06.232375 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.233116 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.234115 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.234455 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.234560 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.235855 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:06.235858 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:06.235860 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:06.235880 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:06.237574 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.238075 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.238079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.238083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.238178 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.238491 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.238496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.238497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.238658 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:06.238769 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:06.239007 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 10:44:06.239164 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":47}
I20260927 10:44:06.239703 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.230686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44553 (local address 127.0.0.1:52506)
0927 10:44:06.231595 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:06.231608 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:06.231695 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:06.231942 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:06.231945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:06.232135 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:06.232246 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.232262 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.233211 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.233214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.233999 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.234009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.234689 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.234693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.234824 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.235316 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:06.235331 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:06.235725 (+ 394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:06.237669 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.237672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.237674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.237982 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.238281 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.238285 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.238287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.238367 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.238750 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:06.238753 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:06.238832 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 10:44:06.239447 (+ 615us) client_negotiation.cc:240] Negotiation successful
0927 10:44:06.239577 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"thread_start_us":77,"threads_started":1}
I20260927 10:44:06.239930 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.230126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.0.1:36638)
0927 10:44:06.230503 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:06.230519 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:06.230669 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:06.231019 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:06.231021 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:06.231228 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:06.231346 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.231351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.232391 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.232394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.232992 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.232999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.233579 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.233584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.233698 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.235039 (+ 1341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:06.235052 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:06.235570 (+ 518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:06.237574 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.237578 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.237580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.237845 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.238136 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.238139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.238141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.238234 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.238731 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:06.238734 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:06.238819 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 10:44:06.238923 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 10:44:06.239317 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":152,"threads_started":1}
Sep 27 10:44:06 dist-test-slave-dzhw krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 10:44:06.253937 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.244382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52520 (local address 127.0.98.61:44553)
0927 10:44:06.244501 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:06.244507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:06.244792 (+ 285us) server_negotiation.cc:407] Connection header received
0927 10:44:06.245005 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:06.245008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:06.245061 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:06.245143 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:06.245745 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.246230 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.247187 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.247465 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.247559 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.250179 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:06.250183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:06.250186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:06.250211 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:06.251897 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.252499 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.252502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.252504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.252587 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.252905 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.252908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.252910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.253122 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:06.253226 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:06.253654 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 10:44:06.253775 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:44:06.254099 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.244349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44553 (local address 127.0.0.1:52520)
0927 10:44:06.244679 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:06.244717 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:06.244863 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:06.245173 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:06.245179 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:06.245533 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:06.245630 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.245635 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.246336 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.246341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.247036 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.247044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.247558 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.247562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.247950 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.248545 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:06.248557 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:06.250031 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:06.252019 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.252032 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.252042 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.252379 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.252685 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.252688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.252691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.252805 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.253241 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:06.253250 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:06.253413 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 10:44:06.253596 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 10:44:06.253855 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":93,"threads_started":1}
I20260927 10:44:06.255714 6491 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:44553: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 44553
}
attrs {
promote: true
}
}
dest_uuid: "ed2c254b7be546a19bad6932d621df69"
cas_config_opid_index: -1
I20260927 10:44:06.256433 6491 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 10:44:06.258198 7051 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 258937809ebb4a2098f5a0f9d45a9ecb (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: true } } }
I20260927 10:44:06.259016 6997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260927 10:44:06.260345 7051 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:44553
I20260927 10:44:06.260353 6997 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 1 FOLLOWER]: Refusing update from remote peer ed2c254b7be546a19bad6932d621df69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 10:44:06.260541 7051 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ed2c254b7be546a19bad6932d621df69" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: true } } }
I20260927 10:44:06.260634 7051 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:06.260962 6955 master.cc:537] Master@127.0.98.61:44553 shutting down...
I20260927 10:44:06.261395 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 10:44:06.263696 6478 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 10:44:06.265151 6478 consensus_peers.cc:487] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 -> Peer 258937809ebb4a2098f5a0f9d45a9ecb (127.0.98.61:44553): Couldn't send request to peer 258937809ebb4a2098f5a0f9d45a9ecb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 10:44:06.265854 6955 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:06.266134 6955 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:06.266244 6955 tablet_replica.cc:330] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260927 10:44:06.279613 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44843
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34339,127.0.98.61:44553,127.0.98.60:44843
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:44:06.281291 6955 master.cc:552] Master@127.0.98.61:44553 shutdown complete.
I20260927 10:44:06.281351 6955 master_runner.cc:305] Clearing existing system tablet
I20260927 10:44:06.281599 6955 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:06.281652 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:06.283066 6955 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 10:44:06.284183 7057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:06.284394 6955 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:06.284518 6955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 10:44:06.284551 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/wal
uuid: "258937809ebb4a2098f5a0f9d45a9ecb"
format_stamp: "Formatted at 2026-09-27 10:44:06 on dist-test-slave-dzhw"
server_key: "d5f2148f36e74e2675f275474269fd3f"
server_key_iv: "253899e787cc5647d95e9277d7acb14a"
server_key_version: "encryptionkey@0"
I20260927 10:44:06.286779 6955 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:06.289785 6955 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 10:44:06.289883 6955 log.cc:1196] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 10:44:06.290150 6955 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb: Deleting consensus metadata
I20260927 10:44:06.290324 6955 master_runner.cc:315] Copying system tablet from 127.0.98.62:34339
I20260927 10:44:06.291539 6955 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:34339
I20260927 10:44:06.301031 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.292201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36652 (local address 127.0.98.62:34339)
0927 10:44:06.292338 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:06.292343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:06.292869 (+ 526us) server_negotiation.cc:407] Connection header received
0927 10:44:06.292973 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:06.292977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:06.293031 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:06.293127 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:06.293618 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.294488 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.295695 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.296039 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.296153 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.297490 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:06.297494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:06.297497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:06.297524 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:06.299269 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.299746 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.299749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.299751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.299841 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.300128 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.300131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.300133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.300360 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:06.300457 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:06.300744 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 10:44:06.300867 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:44:06.301051 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.292459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.0.1:36652)
0927 10:44:06.292741 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:06.292767 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:06.292888 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:06.293149 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:06.293152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:06.293369 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:06.293500 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.293506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.294597 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.294601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.295565 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.295575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.296155 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.296158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.296253 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.296863 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:06.296884 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:06.297341 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:06.299385 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.299389 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.299391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.299646 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.299937 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.299940 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.299942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.300029 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.300456 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:06.300459 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:06.300568 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 10:44:06.300674 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 10:44:06.300839 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":89,"threads_started":1}
I20260927 10:44:06.302135 6511 tablet_copy_service.cc:140] P ed2c254b7be546a19bad6932d621df69: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 258937809ebb4a2098f5a0f9d45a9ecb ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:36652)
I20260927 10:44:06.302330 6511 tablet_copy_service.cc:161] P ed2c254b7be546a19bad6932d621df69: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 258937809ebb4a2098f5a0f9d45a9ecb at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:36652: session id = 258937809ebb4a2098f5a0f9d45a9ecb-00000000000000000000000000000000
I20260927 10:44:06.304315 6511 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 10:44:06.305974 6955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 10:44:06.308403 6955 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb: tablet copy: Starting download of 0 data blocks...
I20260927 10:44:06.308594 6955 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb: tablet copy: Starting download of 1 WAL segments...
I20260927 10:44:06.310405 6955 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 10:44:06.312378 6511 tablet_copy_service.cc:342] P ed2c254b7be546a19bad6932d621df69: Request end of tablet copy session 258937809ebb4a2098f5a0f9d45a9ecb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:36652
I20260927 10:44:06.312505 6511 tablet_copy_service.cc:433] P ed2c254b7be546a19bad6932d621df69: ending tablet copy session 258937809ebb4a2098f5a0f9d45a9ecb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 258937809ebb4a2098f5a0f9d45a9ecb
I20260927 10:44:06.320441 6955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:06.322474 7064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:06.322606 7067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:06.322762 7065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:06.323101 6955 server_base.cc:1034] running on GCE node
I20260927 10:44:06.323259 6955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:06.323550 6955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:06.324697 6955 hybrid_clock.cc:648] HybridClock initialized: now 1790505846324686 us; error 28 us; skew 500 ppm
Sep 27 10:44:06 dist-test-slave-dzhw krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:06.326359 6955 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 10:44:06.326859 6955 webserver.cc:460] Webserver started at http://127.0.98.61:44721/ using document root <none> and password file <none>
I20260927 10:44:06.327051 6955 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:06.327095 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:06.328614 6955 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 10:44:06.329677 7074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:06.329957 6955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 10:44:06.330030 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/wal
uuid: "258937809ebb4a2098f5a0f9d45a9ecb"
format_stamp: "Formatted at 2026-09-27 10:44:06 on dist-test-slave-dzhw"
server_key: "d5f2148f36e74e2675f275474269fd3f"
server_key_iv: "253899e787cc5647d95e9277d7acb14a"
server_key_version: "encryptionkey@0"
I20260927 10:44:06.330111 6955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:06.347596 6955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:06.349799 6955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:06.355131 6955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44553
I20260927 10:44:06.355252 7126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44553 every 8 connection(s)
I20260927 10:44:06.355736 6955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-1/data/info.pb
I20260927 10:44:06.357487 7127 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:44:06.358561 7127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb: Bootstrap starting.
I20260927 10:44:06.368901 7127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:44:06.369249 7127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb: Bootstrap complete.
I20260927 10:44:06.369872 7127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: true } }
I20260927 10:44:06.370034 7127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 258937809ebb4a2098f5a0f9d45a9ecb, State: Initialized, Role: LEARNER
I20260927 10:44:06.370136 7127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: true } }
I20260927 10:44:06.370497 7130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: true } } }
I20260927 10:44:06.370592 7127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:44:06.370692 7130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [sys.catalog]: This master's current role is: LEARNER
I20260927 10:44:06.373799 7141 catalog_manager.cc:1259] Loaded cluster ID: 8d324f4216d949b7b8161f188793fea6
I20260927 10:44:06.373862 7141 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb: loading cluster ID for follower catalog manager: success
I20260927 10:44:06.376020 7141 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb: acquiring CA information for follower catalog manager: success
I20260927 10:44:06.377419 7141 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 10:44:06.414215 7055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:06.414592 7055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:06.414673 7055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:06.418841 7055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:44:06.418923 7055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:06.418960 7055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:06.419016 7055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:44:06.419054 7055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:44:06.424542 7055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34339,127.0.98.61:44553,127.0.98.60:44843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:06.425654 7055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:06.426798 7055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:06.435886 7149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:06.435935 7146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:06.436216 7055 server_base.cc:1034] running on GCE node
W20260927 10:44:06.435886 7147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:06.436898 7055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:06.438009 7055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:06.439158 7055 hybrid_clock.cc:648] HybridClock initialized: now 1790505846439146 us; error 33 us; skew 500 ppm
Sep 27 10:44:06 dist-test-slave-dzhw krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:06.441895 7055 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 10:44:06.442830 7055 webserver.cc:460] Webserver started at http://127.0.98.60:43911/ using document root <none> and password file <none>
I20260927 10:44:06.443351 7055 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:06.443408 7055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:06.443572 7055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:06.445570 7055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/data/instance:
uuid: "fec063b8e3804a5588d34385291a218b"
format_stamp: "Formatted at 2026-09-27 10:44:06 on dist-test-slave-dzhw"
server_key: "53ecae4544b6ec14633efb794f44e776"
server_key_iv: "ab32c5bdb08aa51209a4e3704e97eea0"
server_key_version: "encryptionkey@0"
I20260927 10:44:06.446061 7055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/wal/instance:
uuid: "fec063b8e3804a5588d34385291a218b"
format_stamp: "Formatted at 2026-09-27 10:44:06 on dist-test-slave-dzhw"
server_key: "53ecae4544b6ec14633efb794f44e776"
server_key_iv: "ab32c5bdb08aa51209a4e3704e97eea0"
server_key_version: "encryptionkey@0"
I20260927 10:44:06.449801 7055 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:44:06.452340 7156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:06.453652 7055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:44:06.453807 7055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/wal
uuid: "fec063b8e3804a5588d34385291a218b"
format_stamp: "Formatted at 2026-09-27 10:44:06 on dist-test-slave-dzhw"
server_key: "53ecae4544b6ec14633efb794f44e776"
server_key_iv: "ab32c5bdb08aa51209a4e3704e97eea0"
server_key_version: "encryptionkey@0"
I20260927 10:44:06.453924 7055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:06.475183 7055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:06.478694 7055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:06.478906 7055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:06.487690 7055 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44843
I20260927 10:44:06.488024 7208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44843 every 8 connection(s)
I20260927 10:44:06.488704 7055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/data/info.pb
I20260927 10:44:06.491410 7209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:06.495030 7209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:44:06.497241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 7055
Sep 27 10:44:06 dist-test-slave-dzhw krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 10:44:06 dist-test-slave-dzhw krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 10:44:06.507292 6682 heartbeater.cc:503] Master 127.0.98.62:34339 was elected leader, sending a full tablet report...
I20260927 10:44:06.509459 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.496941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36658 (local address 127.0.98.62:34339)
0927 10:44:06.497128 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:06.497135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:06.497910 (+ 775us) server_negotiation.cc:407] Connection header received
0927 10:44:06.499032 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:06.499036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:06.499097 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:06.499191 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:06.500097 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.500600 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.501690 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.501945 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.502024 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.504913 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:06.504917 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:06.504920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:06.504951 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:06.506586 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.507196 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.507200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.507202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.507292 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.508029 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.508033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.508035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.508300 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:06.508391 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:06.509125 (+ 734us) server_negotiation.cc:299] Negotiation successful
0927 10:44:06.509282 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 10:44:06.510260 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.498082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50038 (local address 127.0.98.60:44843)
0927 10:44:06.498350 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:06.498360 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:06.498390 (+ 30us) server_negotiation.cc:407] Connection header received
0927 10:44:06.498840 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:06.498859 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:06.499042 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:06.499220 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:06.499895 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.500886 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.501750 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.502318 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.502438 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.505297 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:06.505305 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:06.505321 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:06.505351 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:06.507303 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.507958 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.507963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.507965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.508058 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.508360 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.508363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.508365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.508776 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:06.508933 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:06.509195 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 10:44:06.509701 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":410,"server-negotiator.queue_time_us":177,"thread_start_us":83,"threads_started":1}
I20260927 10:44:06.510259 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.497198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.0.1:36658)
0927 10:44:06.497727 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:06.497788 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:06.498830 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:06.499267 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:06.499274 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:06.499671 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:06.499961 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.499972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.500721 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.500725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.501556 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.501564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.502080 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.502083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.502376 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.503022 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:06.503037 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:06.504768 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:06.506693 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.506701 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.506713 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.507071 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.507779 (+ 708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.507782 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.507784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.507924 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.508470 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:06.508476 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:06.508801 (+ 325us) client_negotiation.cc:769] Sending connection context
0927 10:44:06.509038 (+ 237us) client_negotiation.cc:240] Negotiation successful
0927 10:44:06.509374 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":92,"spinlock_wait_cycles":4480,"thread_start_us":145,"threads_started":1}
I20260927 10:44:06.511904 7209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:44:06 dist-test-slave-dzhw krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 10:44:06.519881 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.512394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44553 (local address 127.0.0.1:52526)
0927 10:44:06.512505 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:06.512521 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:06.512586 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:06.513022 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:06.513025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:06.513158 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:06.513269 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.513274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.514143 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.514146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.514774 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.514783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.515197 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.515199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.515273 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.515691 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:06.515702 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:06.516907 (+ 1205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:06.518686 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.518688 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.518690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.518876 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.519105 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.519107 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.519108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.519197 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.519513 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:06.519515 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:06.519588 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 10:44:06.519666 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 10:44:06.519764 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 10:44:06.519979 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.512509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52526 (local address 127.0.98.61:44553)
0927 10:44:06.512813 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:06.512817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:06.512830 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:44:06.512873 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:06.512875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:06.512912 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:06.512982 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:06.513371 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.514023 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.514887 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.515100 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.515179 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.517041 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:06.517044 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:06.517046 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:06.517069 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:06.518576 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.518959 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.518962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.518963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.519030 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.519277 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.519279 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.519280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.519438 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:06.519509 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:06.519726 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 10:44:06.519818 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":134,"threads_started":1}
I20260927 10:44:06.520666 7209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:44:06 dist-test-slave-dzhw krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 10:44:06.528147 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.521127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44843 (local address 127.0.0.1:50044)
0927 10:44:06.521237 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:06.521253 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:06.521322 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:06.521490 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:06.521492 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:06.521613 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:06.521691 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.521695 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.522351 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.522353 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.522903 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.522911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.523313 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.523316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.523371 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.523779 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:06.523789 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:06.524957 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:06.526728 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.526731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.526732 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.526956 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.527227 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.527229 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.527230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.527305 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.527722 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:06.527724 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:06.527796 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 10:44:06.527878 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 10:44:06.527987 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 10:44:06.528168 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.521223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50044 (local address 127.0.98.60:44843)
0927 10:44:06.521328 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:06.521331 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:06.521342 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:44:06.521370 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:06.521372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:06.521402 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:06.521464 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:06.521777 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.522261 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.522996 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.523216 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.523291 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.525074 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:06.525077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:06.525079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:06.525099 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:06.526628 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.527041 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.527043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.527045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.527123 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.527412 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.527415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.527416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.527586 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:06.527740 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:06.527920 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 10:44:06.528046 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":41}
I20260927 10:44:06.532145 7209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b: Bootstrap starting.
I20260927 10:44:06.534354 7209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:06.535096 7209 log.cc:824] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:06.536906 7209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b: No bootstrap required, opened a new log
I20260927 10:44:06.539896 7209 raft_consensus.cc:348] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } }
I20260927 10:44:06.540138 7209 raft_consensus.cc:374] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:06.540222 7209 raft_consensus.cc:729] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec063b8e3804a5588d34385291a218b, State: Initialized, Role: FOLLOWER
I20260927 10:44:06.540657 7209 consensus_queue.cc:260] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } }
I20260927 10:44:06.541118 7215 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } } }
I20260927 10:44:06.541237 7215 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:44:06.541810 7209 sys_catalog.cc:564] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [sys.catalog]: configured and running, proceeding with master startup.
W20260927 10:44:06.546725 7226 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:44:06.546823 7226 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:44:06.550598 7055 master_runner.cc:428] Detected that this master fec063b8e3804a5588d34385291a218b is joining an existing cluster
I20260927 10:44:06.550679 7055 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:44843
I20260927 10:44:06.562167 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.553398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52534 (local address 127.0.98.61:44553)
0927 10:44:06.553525 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:06.553529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:06.554013 (+ 484us) server_negotiation.cc:407] Connection header received
0927 10:44:06.554180 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:06.554184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:06.554245 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:06.554298 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:06.555033 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.555833 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.557012 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.557379 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.557476 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.559290 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:06.559293 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:06.559294 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:06.559320 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:06.560645 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.561109 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.561112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.561113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.561182 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.561451 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.561453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.561454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.561617 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:06.561696 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:06.561947 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 10:44:06.562050 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:44:06.562245 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.553348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44553 (local address 127.0.0.1:52534)
0927 10:44:06.553935 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:06.553948 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:06.554074 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:06.554475 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:06.554478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:06.554731 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:06.554892 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.554903 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.555977 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.555982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.556874 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.556885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.557684 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.557687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.557775 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.558597 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:06.558611 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:06.559141 (+ 530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:06.560750 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.560754 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.560760 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.561014 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.561272 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.561275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.561277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.561367 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.561699 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:06.561702 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:06.561798 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 10:44:06.561903 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 10:44:06.562042 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":22,"thread_start_us":77,"threads_started":1}
I20260927 10:44:06.563895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 6553
I20260927 10:44:06.569821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38249
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34339
--webserver_port=46001
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34339,127.0.98.61:44553
--tserver_master_addrs=127.0.98.62:34339,127.0.98.61:44553,127.0.98.60:44843 with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:44:06.579842 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.553729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50046 (local address 127.0.98.60:44843)
0927 10:44:06.553822 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:06.553828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:06.554082 (+ 254us) server_negotiation.cc:407] Connection header received
0927 10:44:06.554227 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:06.554230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:06.554271 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:06.554315 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:06.554914 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.555619 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.557572 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.557855 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.557917 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.576002 (+ 18085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:06.576006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:06.576009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:06.576029 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:06.578089 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.578642 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.578645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.578647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.578714 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.578952 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.578955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.578956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.579155 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:06.579239 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:06.579551 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 10:44:06.579670 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 10:44:06.580492 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.552977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36674 (local address 127.0.98.62:34339)
0927 10:44:06.553109 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:06.553115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:06.553726 (+ 611us) server_negotiation.cc:407] Connection header received
0927 10:44:06.553909 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:06.553914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:06.553966 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:06.554046 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:06.554760 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.555602 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.557288 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.557644 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.557740 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.576076 (+ 18336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:06.576084 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:06.576091 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:06.576130 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:06.577897 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.578748 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.578754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.578759 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.578854 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.579202 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.579208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.579211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.579565 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:06.579725 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:06.580129 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 10:44:06.580268 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:44:06.580910 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.553659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44843 (local address 127.0.0.1:50046)
0927 10:44:06.554018 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:06.554031 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:06.554126 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:06.554458 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:06.554461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:06.554674 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:06.554817 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.554825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.555734 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.555738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.557472 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.557481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.558045 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.558051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.558219 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.575195 (+ 16976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:06.575212 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:06.575825 (+ 613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:06.578210 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.578213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.578215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.578544 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.578804 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.578807 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.578809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.578877 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.579278 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:06.579281 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:06.579405 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 10:44:06.580376 (+ 971us) client_negotiation.cc:240] Negotiation successful
0927 10:44:06.580736 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":71,"spinlock_wait_cycles":3456,"thread_start_us":75,"threads_started":1}
I20260927 10:44:06.581144 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.552885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.0.1:36674)
0927 10:44:06.553621 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:06.553652 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:06.553800 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:06.554188 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:06.554191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:06.554458 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:06.554612 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.554622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.555744 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.555747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.557141 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.557150 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.558072 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.558074 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.558216 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.575097 (+ 16881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:06.575125 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:06.575850 (+ 725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:06.578061 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.578067 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.578072 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.578603 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.578987 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.578990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.578992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.579086 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.579790 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:06.579795 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:06.579916 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 10:44:06.580660 (+ 744us) client_negotiation.cc:240] Negotiation successful
0927 10:44:06.580841 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":44,"spinlock_wait_cycles":6656,"thread_start_us":115,"threads_started":1}
Sep 27 10:44:06 dist-test-slave-dzhw krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 10:44:06.594493 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.585264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44843 (local address 127.0.0.1:50056)
0927 10:44:06.585381 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:06.585395 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:06.585477 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:06.585779 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:06.585782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:06.585971 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:06.586075 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.586082 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.587137 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.587141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.588040 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.588050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.588635 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.588640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.588756 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.589256 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:06.589270 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:06.590820 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:06.592799 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.592802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.592805 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.593051 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.593343 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.593346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.593348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.593428 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.593943 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:06.593946 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:06.594044 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 10:44:06.594134 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 10:44:06.594280 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 10:44:06.595276 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.585313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50056 (local address 127.0.98.60:44843)
0927 10:44:06.585415 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:06.585421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:06.585435 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:44:06.585612 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:06.585615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:06.585668 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:06.585751 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:06.586212 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.587008 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.588158 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.588502 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.588602 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.590977 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:06.590981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:06.590984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:06.591012 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:06.592657 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.593159 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.593162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.593164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.593246 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.593520 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.593522 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.593524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.593748 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:06.594488 (+ 740us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:06.594992 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 10:44:06.595114 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 10:44:06.596208 6491 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:44843: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fec063b8e3804a5588d34385291a218b"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 44843
}
attrs {
promote: true
}
}
dest_uuid: "ed2c254b7be546a19bad6932d621df69"
cas_config_opid_index: 5
I20260927 10:44:06.596809 6491 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: true } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: true } }
I20260927 10:44:06.597860 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fec063b8e3804a5588d34385291a218b (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: true } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: true } } }
I20260927 10:44:06.598403 7183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:06.599234 7050 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:44843
I20260927 10:44:06.599409 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ed2c254b7be546a19bad6932d621df69" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: true } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: true } } }
I20260927 10:44:06.599519 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:06.599818 7183 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [term 1 FOLLOWER]: Refusing update from remote peer ed2c254b7be546a19bad6932d621df69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 10:44:06.600116 7055 master.cc:537] Master@127.0.98.60:44843 shutting down...
I20260927 10:44:06.600502 7051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 10:44:06.602545 6478 consensus_peers.cc:487] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 -> Peer fec063b8e3804a5588d34385291a218b (127.0.98.60:44843): Couldn't send request to peer fec063b8e3804a5588d34385291a218b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 10:44:06.604990 7055 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:06.605262 7055 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:06.605365 7055 tablet_replica.cc:330] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b: stopping tablet replica
I20260927 10:44:06.605628 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.599688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52536 (local address 127.0.98.61:44553)
0927 10:44:06.599822 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:06.599827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:06.599841 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:44:06.599895 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:06.599897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:06.599963 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:06.600056 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:06.601469 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.602291 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.603977 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.604578 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.604679 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.604827 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:06.605437 (+ 610us) server_negotiation.cc:299] Negotiation successful
0927 10:44:06.605509 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:44:06.606033 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.597920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44553 (local address 127.0.0.1:52536)
0927 10:44:06.598194 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:06.598209 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:06.598300 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:06.600261 (+ 1961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:06.600264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:06.600286 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:06.600389 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.600397 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.602444 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.602447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.603820 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.603831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.604953 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.604957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.605045 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.605166 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 10:44:06.605239 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 10:44:06.605914 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260927 10:44:06.606940 7101 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 1 LEARNER]: Refusing update from remote peer ed2c254b7be546a19bad6932d621df69: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 10:44:06.607438 7051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 10:44:06.609294 7101 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fec063b8e3804a5588d34385291a218b (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: true } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: true } } }
I20260927 10:44:06.611173 7130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed2c254b7be546a19bad6932d621df69. Latest consensus state: current_term: 1 leader_uuid: "ed2c254b7be546a19bad6932d621df69" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: true } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: true } } }
I20260927 10:44:06.611276 7130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [sys.catalog]: This master's current role is: LEARNER
I20260927 10:44:06.611346 7130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed2c254b7be546a19bad6932d621df69" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: true } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: true } } }
I20260927 10:44:06.611400 7130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [sys.catalog]: This master's current role is: LEARNER
I20260927 10:44:06.611663 7055 master.cc:552] Master@127.0.98.60:44843 shutdown complete.
I20260927 10:44:06.611714 7055 master_runner.cc:305] Clearing existing system tablet
I20260927 10:44:06.611878 7051 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69: attempting to promote NON_VOTER 258937809ebb4a2098f5a0f9d45a9ecb to VOTER
I20260927 10:44:06.611963 7055 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:06.612008 7055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:06.612288 7051 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: true } }
I20260927 10:44:06.613433 7055 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 10:44:06.613655 7100 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 1 LEARNER]: Refusing update from remote peer ed2c254b7be546a19bad6932d621df69: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 10:44:06.614305 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 10:44:06.614581 7240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:06.614802 7055 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:06.614909 7055 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 10:44:06.614940 7055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/wal
uuid: "fec063b8e3804a5588d34385291a218b"
format_stamp: "Formatted at 2026-09-27 10:44:06 on dist-test-slave-dzhw"
server_key: "53ecae4544b6ec14633efb794f44e776"
server_key_iv: "ab32c5bdb08aa51209a4e3704e97eea0"
server_key_version: "encryptionkey@0"
I20260927 10:44:06.615535 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.613237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44843 (local address 127.0.0.1:50066)
0927 10:44:06.615372 (+ 2135us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:06.615451 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2053}
I20260927 10:44:06.616339 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.615989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44843 (local address 127.0.0.1:50068)
0927 10:44:06.616218 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:06.616265 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
W20260927 10:44:06.616411 6478 consensus_peers.cc:487] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 -> Peer fec063b8e3804a5588d34385291a218b (127.0.98.60:44843): Couldn't send request to peer fec063b8e3804a5588d34385291a218b. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 10:44:06.617041 7055 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:06.617362 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 258937809ebb4a2098f5a0f9d45a9ecb (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: true } } }
I20260927 10:44:06.618754 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ed2c254b7be546a19bad6932d621df69" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: true } } }
I20260927 10:44:06.618863 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:06.618999 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed2c254b7be546a19bad6932d621df69" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: true } } }
I20260927 10:44:06.619083 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:06.619752 7100 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 258937809ebb4a2098f5a0f9d45a9ecb (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: true } } }
I20260927 10:44:06.620122 7055 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 10:44:06.620251 7055 log.cc:1196] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 10:44:06.620556 7055 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b: Deleting consensus metadata
I20260927 10:44:06.620769 7055 master_runner.cc:315] Copying system tablet from 127.0.98.62:34339
I20260927 10:44:06.621821 7237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed2c254b7be546a19bad6932d621df69" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: true } } }
I20260927 10:44:06.621946 7237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:44:06.621896 7055 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:34339
I20260927 10:44:06.631289 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.622578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.0.1:36686)
0927 10:44:06.622856 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:06.622872 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:06.622979 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:06.623247 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:06.623250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:06.623476 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:06.623597 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.623604 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.624652 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.624655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.625529 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.625539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.626120 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.626123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.626211 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.626784 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:06.626801 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:06.627252 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:06.629491 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.629495 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.629516 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.629771 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.630092 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.630095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.630097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.630191 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.630721 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:06.630724 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:06.630824 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 10:44:06.630931 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 10:44:06.631077 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":68,"threads_started":1}
I20260927 10:44:06.631319 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.622666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36686 (local address 127.0.98.62:34339)
0927 10:44:06.622781 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:06.622785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:06.622955 (+ 170us) server_negotiation.cc:407] Connection header received
0927 10:44:06.623068 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:06.623071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:06.623120 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:06.623203 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:06.623717 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.624528 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.625654 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.625997 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.626100 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.627401 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:06.627405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:06.627408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:06.627434 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:06.629371 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.629906 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.629909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.629911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.629997 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.630272 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.630275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.630277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.630577 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:06.630716 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:06.630999 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 10:44:06.631113 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 10:44:06.632145 6511 tablet_copy_service.cc:140] P ed2c254b7be546a19bad6932d621df69: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fec063b8e3804a5588d34385291a218b ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:36686)
I20260927 10:44:06.632248 6511 tablet_copy_service.cc:161] P ed2c254b7be546a19bad6932d621df69: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fec063b8e3804a5588d34385291a218b at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:36686: session id = fec063b8e3804a5588d34385291a218b-00000000000000000000000000000000
I20260927 10:44:06.633456 6511 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 10:44:06.634920 7055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 10:44:06.637190 7055 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b: tablet copy: Starting download of 0 data blocks...
I20260927 10:44:06.637377 7055 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b: tablet copy: Starting download of 1 WAL segments...
I20260927 10:44:06.638921 7055 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 10:44:06.640676 6511 tablet_copy_service.cc:342] P ed2c254b7be546a19bad6932d621df69: Request end of tablet copy session fec063b8e3804a5588d34385291a218b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:36686
I20260927 10:44:06.640769 6511 tablet_copy_service.cc:433] P ed2c254b7be546a19bad6932d621df69: ending tablet copy session fec063b8e3804a5588d34385291a218b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fec063b8e3804a5588d34385291a218b
I20260927 10:44:06.650969 7055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:06.652715 7251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:06.652827 7249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:06.652967 7248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:06.654613 7055 server_base.cc:1034] running on GCE node
I20260927 10:44:06.654786 7055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:06.655071 7055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:06.656222 7055 hybrid_clock.cc:648] HybridClock initialized: now 1790505846656209 us; error 30 us; skew 500 ppm
Sep 27 10:44:06 dist-test-slave-dzhw krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:06.657876 7055 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 10:44:06.658336 7055 webserver.cc:460] Webserver started at http://127.0.98.60:39393/ using document root <none> and password file <none>
I20260927 10:44:06.658565 7055 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:06.658615 7055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:06.659958 7055 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 10:44:06.660956 7258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:06.661233 7055 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 10:44:06.661306 7055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/wal
uuid: "fec063b8e3804a5588d34385291a218b"
format_stamp: "Formatted at 2026-09-27 10:44:06 on dist-test-slave-dzhw"
server_key: "53ecae4544b6ec14633efb794f44e776"
server_key_iv: "ab32c5bdb08aa51209a4e3704e97eea0"
server_key_version: "encryptionkey@0"
I20260927 10:44:06.661391 7055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:06.676210 7055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:06.678622 7055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:06.684868 7055 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44843
I20260927 10:44:06.684890 7310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44843 every 8 connection(s)
I20260927 10:44:06.685577 7055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/master-2/data/info.pb
I20260927 10:44:06.687474 7311 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:44:06.688661 7311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b: Bootstrap starting.
I20260927 10:44:06.700663 7311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:44:06.700997 7311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b: Bootstrap complete.
I20260927 10:44:06.701879 7311 raft_consensus.cc:348] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: true } }
I20260927 10:44:06.702034 7311 raft_consensus.cc:729] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fec063b8e3804a5588d34385291a218b, State: Initialized, Role: LEARNER
I20260927 10:44:06.702152 7311 consensus_queue.cc:260] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: true } }
I20260927 10:44:06.702553 7314 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: true } } }
I20260927 10:44:06.702685 7311 sys_catalog.cc:564] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:44:06.702756 7314 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [sys.catalog]: This master's current role is: LEARNER
I20260927 10:44:06.705576 7325 catalog_manager.cc:1259] Loaded cluster ID: 8d324f4216d949b7b8161f188793fea6
I20260927 10:44:06.705703 7325 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b: loading cluster ID for follower catalog manager: success
I20260927 10:44:06.707958 7325 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b: acquiring CA information for follower catalog manager: success
I20260927 10:44:06.709371 7325 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 10:44:06.721787 7235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:06.722066 7235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:06.722155 7235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:06.726308 7235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:06.726413 7235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:06.726538 7235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:44:06.732856 7235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38249
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46001
--tserver_master_addrs=127.0.98.62:34339,127.0.98.61:44553,127.0.98.60:44843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:06.733992 7235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:06.735069 7235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:44:06.743044 6816 heartbeater.cc:503] Master 127.0.98.62:34339 was elected leader, sending a full tablet report...
W20260927 10:44:06.744843 7331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:06.744853 7333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:06.744848 7330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:06.745225 7235 server_base.cc:1034] running on GCE node
I20260927 10:44:06.745615 7235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:06.746276 7235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:06.747464 7235 hybrid_clock.cc:648] HybridClock initialized: now 1790505846747453 us; error 35 us; skew 500 ppm
Sep 27 10:44:06 dist-test-slave-dzhw krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:06.750078 7235 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 10:44:06.751008 7235 webserver.cc:460] Webserver started at http://127.0.98.1:46001/ using document root <none> and password file <none>
I20260927 10:44:06.751572 7235 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:06.751690 7235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:06.755503 7235 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 10:44:06.757535 7340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:06.758957 7235 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:44:06.759111 7235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "e5c1e1840c784fdd871dc6b3572c1e83"
format_stamp: "Formatted at 2026-09-27 10:44:05 on dist-test-slave-dzhw"
server_key: "d208e4876c4cdf3873875961de6bfaf3"
server_key_iv: "8ade8502c08046333b8e2ac710be661d"
server_key_version: "encryptionkey@0"
I20260927 10:44:06.759690 7235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:06.779589 7235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:06.782744 7235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:06.782949 7235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:06.783645 7235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:06.784516 7235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:06.784588 7235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:06.784652 7235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:06.784705 7235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:06.794055 7235 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38249
I20260927 10:44:06.794078 7453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38249 every 8 connection(s)
I20260927 10:44:06.795064 7235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:44:06.801558 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 7235
I20260927 10:44:06.801746 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 6687
Sep 27 10:44:06 dist-test-slave-dzhw krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 10:44:06 dist-test-slave-dzhw krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 10:44:06 dist-test-slave-dzhw krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 10:44:06.810835 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44599
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34339
--webserver_port=38753
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34339,127.0.98.61:44553
--tserver_master_addrs=127.0.98.62:34339,127.0.98.61:44553,127.0.98.60:44843 with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:44:06.814016 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.796811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35523 (local address 127.0.98.62:34339)
0927 10:44:06.796948 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:06.796952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:06.798159 (+ 1207us) server_negotiation.cc:407] Connection header received
0927 10:44:06.799322 (+ 1163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:06.799325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:06.799384 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:06.799468 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:06.801091 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.801925 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.804075 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.804417 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.804491 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.809619 (+ 5128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:06.809624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:06.809627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:06.809662 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:06.811564 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.812125 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.812128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.812130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.812211 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.812563 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.812567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.812569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.812809 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:06.812891 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:06.813712 (+ 821us) server_negotiation.cc:299] Negotiation successful
0927 10:44:06.813844 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:44:06.814016 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.798633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33751 (local address 127.0.98.61:44553)
0927 10:44:06.798765 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:06.798770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:06.798915 (+ 145us) server_negotiation.cc:407] Connection header received
0927 10:44:06.799522 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:06.799525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:06.799581 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:06.799633 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:06.801091 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.801923 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.804075 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.804417 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.804491 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.809084 (+ 4593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:06.809091 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:06.809098 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:06.809135 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:06.810980 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.812046 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.812053 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.812058 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.812150 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.812564 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.812567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.812568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.812806 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:06.812891 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:06.813712 (+ 821us) server_negotiation.cc:299] Negotiation successful
0927 10:44:06.813844 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:44:06.814982 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.799107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39791 (local address 127.0.98.60:44843)
0927 10:44:06.799751 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:06.799757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:06.799772 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:44:06.799818 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:06.799820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:06.799857 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:06.799948 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:06.801081 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.802141 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.804115 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.804677 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.804806 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.809490 (+ 4684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:06.809497 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:06.809503 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:06.809538 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:06.812590 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.813102 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.813107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.813109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.813209 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:06.813526 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:06.813530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:06.813534 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:06.813775 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:06.813869 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:06.814575 (+ 706us) server_negotiation.cc:299] Negotiation successful
0927 10:44:06.814724 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":91,"threads_started":1}
I20260927 10:44:06.815757 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.798516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44553 (local address 127.0.98.1:33751)
0927 10:44:06.798808 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:06.798824 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:06.799404 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:06.799779 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:06.799789 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:06.800429 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:06.800926 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.800945 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.802109 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.802113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.803934 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.803945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.804700 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.804703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.805472 (+ 769us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.806347 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:06.806363 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:06.808870 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:06.811222 (+ 2352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.811231 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.811250 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.811924 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.812262 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.812267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.812272 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.812462 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.812963 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:06.812971 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:06.813408 (+ 437us) client_negotiation.cc:769] Sending connection context
0927 10:44:06.814582 (+ 1174us) client_negotiation.cc:240] Negotiation successful
0927 10:44:06.814696 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":794,"spinlock_wait_cycles":42112,"thread_start_us":99,"threads_started":1}
I20260927 10:44:06.816016 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.797168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.1:35523)
0927 10:44:06.798010 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:06.798050 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:06.799085 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:06.799780 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:06.799789 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:06.800423 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:06.800926 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.800945 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.802109 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.802113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.803934 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.803945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.804700 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.804703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.805510 (+ 807us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.806293 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:06.806319 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:06.809449 (+ 3130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:06.811732 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.811738 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.811742 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.812024 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.812329 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.812332 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.812333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.812453 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.812979 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:06.812981 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:06.813408 (+ 427us) client_negotiation.cc:769] Sending connection context
0927 10:44:06.814235 (+ 827us) client_negotiation.cc:240] Negotiation successful
0927 10:44:06.814582 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":1097,"spinlock_wait_cycles":50944,"thread_start_us":517,"threads_started":1}
I20260927 10:44:06.816453 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:06.798915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44843 (local address 127.0.98.1:39791)
0927 10:44:06.799141 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:06.799153 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:06.799672 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:06.799968 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:06.799971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:06.800403 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:06.800928 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.800942 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.802287 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.802293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.803973 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:06.803987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:06.804907 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:06.804910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:06.805017 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:06.806471 (+ 1454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:06.806484 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:06.809169 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:06.812723 (+ 3554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.812728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.812730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.812983 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.813327 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:06.813330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:06.813332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:06.813420 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:06.813914 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:06.813920 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:06.814038 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 10:44:06.814160 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 10:44:06.816289 (+ 2129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":1867,"spinlock_wait_cycles":896,"thread_start_us":106,"threads_started":1}
I20260927 10:44:06.817533 7454 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44843
I20260927 10:44:06.817615 7455 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34339
I20260927 10:44:06.817803 7455 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:06.817804 7454 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:06.818317 7456 heartbeater.cc:347] Connected to a master server at 127.0.98.61:44553
I20260927 10:44:06.818390 7456 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:06.818522 7454 heartbeater.cc:511] Master 127.0.98.60:44843 requested a full tablet report, sending...
I20260927 10:44:06.818565 7456 heartbeater.cc:511] Master 127.0.98.61:44553 requested a full tablet report, sending...
I20260927 10:44:06.818526 7455 heartbeater.cc:511] Master 127.0.98.62:34339 requested a full tablet report, sending...
I20260927 10:44:06.819967 6491 ts_manager.cc:194] Re-registered known tserver with Master: e5c1e1840c784fdd871dc6b3572c1e83 (127.0.98.1:38249)
I20260927 10:44:06.820123 7275 ts_manager.cc:194] Registered new tserver with Master: e5c1e1840c784fdd871dc6b3572c1e83 (127.0.98.1:38249)
I20260927 10:44:06.820312 7091 ts_manager.cc:194] Registered new tserver with Master: e5c1e1840c784fdd871dc6b3572c1e83 (127.0.98.1:38249)
I20260927 10:44:06.820705 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35523
W20260927 10:44:06.925834 7462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:06.926157 7462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:06.926226 7462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:06.930823 7462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:06.930917 7462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:06.931003 7462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:44:06.936610 7462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38753
--tserver_master_addrs=127.0.98.62:34339,127.0.98.61:44553,127.0.98.60:44843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:06.937665 7462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:06.938730 7462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:06.948257 7468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:06.948257 7470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:06.948274 7467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:06.948714 7462 server_base.cc:1034] running on GCE node
I20260927 10:44:06.949162 7462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:06.949801 7462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:06.950976 7462 hybrid_clock.cc:648] HybridClock initialized: now 1790505846950933 us; error 47 us; skew 500 ppm
I20260927 10:44:06.952704 6950 heartbeater.cc:503] Master 127.0.98.62:34339 was elected leader, sending a full tablet report...
Sep 27 10:44:06 dist-test-slave-dzhw krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:06.954161 7462 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 10:44:06.955085 7462 webserver.cc:460] Webserver started at http://127.0.98.2:38753/ using document root <none> and password file <none>
I20260927 10:44:06.955605 7462 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:06.955686 7462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:06.959573 7462 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 10:44:06.961629 7477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:06.963060 7462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:44:06.963160 7462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "ecaafcecaf3e414984d0c826a0c4246d"
format_stamp: "Formatted at 2026-09-27 10:44:05 on dist-test-slave-dzhw"
server_key: "fd08cb6245077e8fdcd5508114e172d7"
server_key_iv: "4a859f10d455fc8ffe5d3fcdced2a747"
server_key_version: "encryptionkey@0"
I20260927 10:44:06.963713 7462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:06.991912 7462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:06.995015 7462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:06.995173 7462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:06.995697 7462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:06.996505 7462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:06.996551 7462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:06.996590 7462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:06.996606 7462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:07.005708 7462 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44599
I20260927 10:44:07.005733 7590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44599 every 8 connection(s)
I20260927 10:44:07.006798 7462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:44:07.008400 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 7462
I20260927 10:44:07.008623 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 6821
I20260927 10:44:07.016448 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44633
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34339
--webserver_port=43719
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34339,127.0.98.61:44553
--tserver_master_addrs=127.0.98.62:34339,127.0.98.61:44553,127.0.98.60:44843 with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:44:07 dist-test-slave-dzhw krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 10:44:07 dist-test-slave-dzhw krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 10:44:07 dist-test-slave-dzhw krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 10:44:07.027034 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.008714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46977 (local address 127.0.98.60:44843)
0927 10:44:07.009210 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:07.009215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:07.010495 (+ 1280us) server_negotiation.cc:407] Connection header received
0927 10:44:07.011613 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:07.011617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:07.011682 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:07.011739 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:07.013686 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.014554 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.016714 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.017073 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.017140 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.022242 (+ 5102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:07.022246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:07.022249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:07.022281 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:07.024250 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.025082 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.025088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.025093 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.025184 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.025608 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.025611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.025613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.025859 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:07.025929 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:07.026728 (+ 799us) server_negotiation.cc:299] Negotiation successful
0927 10:44:07.026864 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405}
I20260927 10:44:07.027034 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.011756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44111 (local address 127.0.98.61:44553)
0927 10:44:07.011926 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:07.011932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:07.012286 (+ 354us) server_negotiation.cc:407] Connection header received
0927 10:44:07.012465 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:07.012469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:07.012513 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:07.012566 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:07.013686 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.014554 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.016714 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.017046 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.017126 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.022241 (+ 5115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:07.022246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:07.022249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:07.022277 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:07.025047 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.025507 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.025511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.025512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.025599 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.025915 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.025918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.025920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.026136 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:07.026200 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:07.026728 (+ 528us) server_negotiation.cc:299] Negotiation successful
0927 10:44:07.026868 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:44:07.027163 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.011509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56605 (local address 127.0.98.62:34339)
0927 10:44:07.011650 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:07.011656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:07.012209 (+ 553us) server_negotiation.cc:407] Connection header received
0927 10:44:07.012398 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:07.012404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:07.012463 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:07.012522 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:07.013687 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.014726 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.016720 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.017154 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.017249 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.022333 (+ 5084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:07.022339 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:07.022344 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:07.022382 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:07.024447 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.025133 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.025139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.025143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.025241 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.025642 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.025648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.025651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.025932 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:07.026016 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:07.026762 (+ 746us) server_negotiation.cc:299] Negotiation successful
0927 10:44:07.026913 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:44:07.029098 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.011879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44553 (local address 127.0.98.2:44111)
0927 10:44:07.012198 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:07.012211 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:07.012354 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:07.012697 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:07.012699 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:07.012965 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:07.013519 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:07.013537 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.014740 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.014744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:07.016592 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:07.016598 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.017333 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.017337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:07.017642 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.018988 (+ 1346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:07.019002 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:07.022008 (+ 3006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:07.025161 (+ 3153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:07.025164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:07.025167 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:07.025397 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:07.025710 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:07.025713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:07.025715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:07.025796 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:07.026267 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:07.026270 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:07.026396 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 10:44:07.027307 (+ 911us) client_negotiation.cc:240] Negotiation successful
0927 10:44:07.027793 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":702,"spinlock_wait_cycles":8192,"thread_start_us":78,"threads_started":1}
I20260927 10:44:07.029400 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.011358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.2:56605)
0927 10:44:07.012112 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:07.012128 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:07.012283 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:07.012683 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:07.012689 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:07.013011 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:07.013523 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:07.013541 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.014888 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.014894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:07.016584 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:07.016598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.017442 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.017448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:07.017700 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.018976 (+ 1276us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:07.018991 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:07.022130 (+ 3139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:07.024595 (+ 2465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:07.024600 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:07.024603 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:07.025006 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:07.025371 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:07.025377 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:07.025381 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:07.025518 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:07.026102 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:07.026106 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:07.026397 (+ 291us) client_negotiation.cc:769] Sending connection context
0927 10:44:07.027375 (+ 978us) client_negotiation.cc:240] Negotiation successful
0927 10:44:07.027946 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":515,"spinlock_wait_cycles":9728,"thread_start_us":70,"threads_started":1}
I20260927 10:44:07.029109 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.009160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44843 (local address 127.0.98.2:46977)
0927 10:44:07.010268 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:07.010322 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:07.011365 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:07.012173 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:07.012182 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:07.012696 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:07.013521 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:07.013537 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.014740 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.014744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:07.016575 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:07.016587 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.017333 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.017337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:07.017648 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.018627 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:07.018653 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:07.022008 (+ 3355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:07.024420 (+ 2412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:07.024429 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:07.024454 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:07.024951 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:07.025300 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:07.025306 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:07.025308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:07.025504 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:07.026009 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:07.026016 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:07.026395 (+ 379us) client_negotiation.cc:769] Sending connection context
0927 10:44:07.027794 (+ 1399us) client_negotiation.cc:240] Negotiation successful
0927 10:44:07.028069 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":702,"spinlock_wait_cycles":9984,"thread_start_us":723,"threads_started":1}
I20260927 10:44:07.030450 7593 heartbeater.cc:347] Connected to a master server at 127.0.98.61:44553
I20260927 10:44:07.030519 7592 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34339
I20260927 10:44:07.030701 7593 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:07.031304 7593 heartbeater.cc:511] Master 127.0.98.61:44553 requested a full tablet report, sending...
I20260927 10:44:07.031829 7591 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44843
I20260927 10:44:07.031904 7591 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:07.032060 7591 heartbeater.cc:511] Master 127.0.98.60:44843 requested a full tablet report, sending...
I20260927 10:44:07.030704 7592 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:07.032330 7592 heartbeater.cc:511] Master 127.0.98.62:34339 requested a full tablet report, sending...
I20260927 10:44:07.032580 7091 ts_manager.cc:194] Registered new tserver with Master: ecaafcecaf3e414984d0c826a0c4246d (127.0.98.2:44599)
I20260927 10:44:07.032999 7275 ts_manager.cc:194] Registered new tserver with Master: ecaafcecaf3e414984d0c826a0c4246d (127.0.98.2:44599)
I20260927 10:44:07.033521 6491 ts_manager.cc:194] Re-registered known tserver with Master: ecaafcecaf3e414984d0c826a0c4246d (127.0.98.2:44599)
I20260927 10:44:07.034060 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56605
I20260927 10:44:07.090600 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.087195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50082 (local address 127.0.98.60:44843)
0927 10:44:07.087355 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:07.087360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:07.087376 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:44:07.087525 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:07.087528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:07.087597 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:07.087695 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:07.087961 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.088503 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.089564 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.090008 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.090131 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.090290 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:07.090363 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 10:44:07.090452 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 10:44:07.090591 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.087104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44843 (local address 127.0.0.1:50082)
0927 10:44:07.087249 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:07.087268 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:07.087397 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:07.087710 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:07.087713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:07.087731 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:07.087848 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:07.087854 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.088626 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.088631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:07.089434 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:07.089443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.090146 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.090150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:07.090243 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.090265 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:44:07.090340 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 10:44:07.090446 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 10:44:07.091353 7285 raft_consensus.cc:1203] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 10:44:07.091799 7314 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed2c254b7be546a19bad6932d621df69. Latest consensus state: current_term: 1 leader_uuid: "ed2c254b7be546a19bad6932d621df69" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: true } } }
I20260927 10:44:07.091960 7314 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [sys.catalog]: This master's current role is: LEARNER
W20260927 10:44:07.134029 7598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:07.134338 7598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:07.134473 7598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:07.138507 7598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:07.138602 7598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:07.138702 7598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:44:07.144616 7598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43719
--tserver_master_addrs=127.0.98.62:34339,127.0.98.61:44553,127.0.98.60:44843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:07.145706 7598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:07.146781 7598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:07.156627 7603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:07.156706 7606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:07.156627 7604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:07.156766 7598 server_base.cc:1034] running on GCE node
I20260927 10:44:07.157315 7598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:07.157971 7598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:07.159158 7598 hybrid_clock.cc:648] HybridClock initialized: now 1790505847159150 us; error 35 us; skew 500 ppm
Sep 27 10:44:07 dist-test-slave-dzhw krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505847, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:07.161827 7598 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 10:44:07.162842 7598 webserver.cc:460] Webserver started at http://127.0.98.3:43719/ using document root <none> and password file <none>
I20260927 10:44:07.163386 7598 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:07.163452 7598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:07.167208 7598 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 10:44:07.169248 7613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:07.170763 7598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:44:07.170953 7598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "babeb35fec414bac989abcc216118c74"
format_stamp: "Formatted at 2026-09-27 10:44:05 on dist-test-slave-dzhw"
server_key: "d6166917264acbbee766c1148d1ea695"
server_key_iv: "cbfa7252c76fdb43ab3465c22da30b91"
server_key_version: "encryptionkey@0"
I20260927 10:44:07.171548 7598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:07.183457 7598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:07.186738 7598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:07.186952 7598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:07.187582 7598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:07.188464 7598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:07.188539 7598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:07.188618 7598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:07.188663 7598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:07.197795 7598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44633
I20260927 10:44:07.197822 7726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44633 every 8 connection(s)
I20260927 10:44:07.198863 7598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:44:07.203766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 7598
Sep 27 10:44:07 dist-test-slave-dzhw krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505847, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 10:44:07 dist-test-slave-dzhw krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505847, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 10:44:07.217438 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.202581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37731 (local address 127.0.98.61:44553)
0927 10:44:07.202723 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:07.202727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:07.202929 (+ 202us) server_negotiation.cc:407] Connection header received
0927 10:44:07.203240 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:07.203243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:07.203316 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:07.203374 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:07.205329 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.206084 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.208126 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.208461 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.208536 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.212563 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:07.212568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:07.212570 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:07.212598 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:07.214607 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.215211 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.215215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.215217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.215297 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.215663 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.215666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.215667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.215884 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:07.216002 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:07.217136 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0927 10:44:07.217256 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 10:44:07.219514 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.202501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44553 (local address 127.0.98.3:37731)
0927 10:44:07.202844 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:07.202858 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:07.203002 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:07.203897 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:07.203906 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:07.204901 (+ 995us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:07.205146 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:07.205163 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.206485 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.206488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:07.208002 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:07.208011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.208741 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.208747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:07.208979 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.209821 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:07.209835 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:07.212355 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:07.214733 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:07.214740 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:07.214758 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:07.215088 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:07.215390 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:07.215393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:07.215395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:07.215561 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:07.216074 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:07.216081 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:07.216477 (+ 396us) client_negotiation.cc:769] Sending connection context
0927 10:44:07.217862 (+ 1385us) client_negotiation.cc:240] Negotiation successful
0927 10:44:07.218311 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":24,"thread_start_us":120,"threads_started":1}
Sep 27 10:44:07 dist-test-slave-dzhw krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505847, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 10:44:07.221712 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.206233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36698 (local address 127.0.98.62:34339)
0927 10:44:07.210320 (+ 4087us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:07.210324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:07.210338 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:44:07.210373 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:07.210377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:07.210443 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:07.210524 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:07.211501 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.212306 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.214483 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.214926 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.215037 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.217682 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:07.217686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:07.217690 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:07.217721 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:07.219576 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.220097 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.220100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.220102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.220191 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.220572 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.220578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.220581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.220806 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:07.221303 (+ 497us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:07.221381 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 10:44:07.221510 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4004,"thread_start_us":60,"threads_started":1}
I20260927 10:44:07.223263 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.203021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59885 (local address 127.0.98.62:34339)
0927 10:44:07.203149 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:07.203153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:07.203355 (+ 202us) server_negotiation.cc:407] Connection header received
0927 10:44:07.203519 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:07.203522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:07.203568 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:07.203619 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:07.206579 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.207382 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.208525 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.208860 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.208926 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.213666 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:07.213670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:07.213672 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:07.213699 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:07.215390 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.219896 (+ 4506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.219902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.219907 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.220003 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.221477 (+ 1474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.221482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.221484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.221700 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:07.221787 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:07.222968 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0927 10:44:07.223085 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:44:07.223488 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.206857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50086 (local address 127.0.98.60:44843)
0927 10:44:07.207389 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:07.207393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:07.211680 (+ 4287us) server_negotiation.cc:407] Connection header received
0927 10:44:07.211723 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:07.211726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:07.211777 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:07.211852 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:07.212708 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.215905 (+ 3197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.217013 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.217327 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.217418 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.219060 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:07.219064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:07.219067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:07.219092 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:07.221131 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.221633 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.221637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.221638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.221718 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.222017 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.222020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.222022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.222217 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:07.222335 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:07.222896 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 10:44:07.223024 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":69,"threads_started":1}
I20260927 10:44:07.224371 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.206246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52552 (local address 127.0.98.61:44553)
0927 10:44:07.207798 (+ 1552us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:07.207802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:07.208092 (+ 290us) server_negotiation.cc:407] Connection header received
0927 10:44:07.208128 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:07.208131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:07.208176 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:07.208245 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:07.209282 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.210089 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.215632 (+ 5543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.216053 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.216137 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.219305 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:07.219310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:07.219313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:07.219340 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:07.221042 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.221770 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.221774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.221777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.221865 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.222158 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.222161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.222163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.222357 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:07.222493 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:07.222834 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 10:44:07.223174 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1476,"thread_start_us":70,"threads_started":1}
I20260927 10:44:07.224218 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.202923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.3:59885)
0927 10:44:07.203276 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:07.203290 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:07.203419 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:07.203938 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:07.203941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:07.204514 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:07.205145 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:07.205162 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.207516 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.207519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:07.208398 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:07.208407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.209069 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.209072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:07.209182 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.209691 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:07.209711 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:07.213501 (+ 3790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:07.216718 (+ 3217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:07.216720 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:07.216722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:07.216962 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:07.221255 (+ 4293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:07.221258 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:07.221260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:07.221361 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:07.222271 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:07.222276 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:07.222373 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 10:44:07.223827 (+ 1454us) client_negotiation.cc:240] Negotiation successful
0927 10:44:07.223977 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":74,"threads_started":1}
I20260927 10:44:07.225980 7729 heartbeater.cc:347] Connected to a master server at 127.0.98.61:44553
I20260927 10:44:07.226233 7728 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34339
I20260927 10:44:07.226258 7729 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:07.226290 7728 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:07.226876 7729 heartbeater.cc:511] Master 127.0.98.61:44553 requested a full tablet report, sending...
I20260927 10:44:07.227316 7728 heartbeater.cc:511] Master 127.0.98.62:34339 requested a full tablet report, sending...
I20260927 10:44:07.228055 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.201105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44843 (local address 127.0.98.3:43751)
0927 10:44:07.201966 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:07.202007 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:07.203002 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:07.204169 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:07.204173 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:07.205316 (+ 1143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:07.205414 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:07.205419 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.206485 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.206488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:07.211205 (+ 4717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:07.211215 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.216672 (+ 5457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.216677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:07.216805 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.217311 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:07.217325 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:07.222785 (+ 5460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:07.226397 (+ 3612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:07.226400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:07.226403 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:07.226666 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:07.226963 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:07.226966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:07.226968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:07.227048 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:07.227530 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:07.227533 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:07.227613 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 10:44:07.227709 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 10:44:07.227846 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":3787,"thread_start_us":540,"threads_started":1}
I20260927 10:44:07.228070 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.200693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43751 (local address 127.0.98.60:44843)
0927 10:44:07.200855 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:07.200860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:07.202132 (+ 1272us) server_negotiation.cc:407] Connection header received
0927 10:44:07.203259 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:07.203263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:07.203319 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:07.203386 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:07.205531 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.206318 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.216055 (+ 9737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.216402 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.216490 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.223161 (+ 6671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:07.223165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:07.223168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:07.223191 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:07.226273 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.226778 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.226782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.226784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.226862 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.227174 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.227178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.227179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.227386 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:07.227484 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:07.227809 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 10:44:07.227934 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 10:44:07.228240 7090 ts_manager.cc:194] Registered new tserver with Master: babeb35fec414bac989abcc216118c74 (127.0.98.3:44633)
I20260927 10:44:07.228652 7727 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44843
I20260927 10:44:07.228657 6490 ts_manager.cc:194] Re-registered known tserver with Master: babeb35fec414bac989abcc216118c74 (127.0.98.3:44633)
I20260927 10:44:07.228729 7727 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:07.228900 7727 heartbeater.cc:511] Master 127.0.98.60:44843 requested a full tablet report, sending...
I20260927 10:44:07.225715 6491 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 10:44:07.229401 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59885
I20260927 10:44:07.229774 7275 ts_manager.cc:194] Registered new tserver with Master: babeb35fec414bac989abcc216118c74 (127.0.98.3:44633)
W20260927 10:44:07.230310 6491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:44:07.233234 7745 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed2c254b7be546a19bad6932d621df69" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: true } } }
I20260927 10:44:07.233366 7745 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:07.236171 7745 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69: attempting to promote NON_VOTER fec063b8e3804a5588d34385291a218b to VOTER
I20260927 10:44:07.236490 7745 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: false } }
I20260927 10:44:07.237516 7285 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [term 1 LEARNER]: Refusing update from remote peer ed2c254b7be546a19bad6932d621df69: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 10:44:07.237586 7100 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 1 FOLLOWER]: Refusing update from remote peer ed2c254b7be546a19bad6932d621df69: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 10:44:07.237937 7238 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 10:44:07.238229 7238 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 10:44:07.239740 7238 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fec063b8e3804a5588d34385291a218b (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: false } } }
I20260927 10:44:07.241195 7745 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ed2c254b7be546a19bad6932d621df69" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: false } } }
I20260927 10:44:07.241305 7745 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:07.241634 7100 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fec063b8e3804a5588d34385291a218b (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: false } } }
I20260927 10:44:07.241786 7745 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed2c254b7be546a19bad6932d621df69" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: false } } }
I20260927 10:44:07.241891 7745 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:07.242076 7238 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed2c254b7be546a19bad6932d621df69" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: false } } }
I20260927 10:44:07.242208 7238 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69 [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:07.242348 7285 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fec063b8e3804a5588d34385291a218b (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: false } } }
I20260927 10:44:07.243335 7755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed2c254b7be546a19bad6932d621df69" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: false } } }
I20260927 10:44:07.243466 7755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:44:07.244369 7314 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed2c254b7be546a19bad6932d621df69" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: false } } }
I20260927 10:44:07.244480 7314 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:44:07.252683 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.247472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60238 (local address 127.0.98.2:44599)
0927 10:44:07.248386 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:07.248397 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:07.248699 (+ 302us) server_negotiation.cc:407] Connection header received
0927 10:44:07.248765 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:07.248768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:07.248904 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:07.249059 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:07.249234 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.249945 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.251272 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.252109 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.252241 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.252398 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:07.252448 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 10:44:07.252530 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":145,"threads_started":1}
I20260927 10:44:07.252779 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.246808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44633 (local address 127.0.0.1:56056)
0927 10:44:07.246942 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:07.246959 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:07.247067 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:07.248341 (+ 1274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:07.248344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:07.248363 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:07.248482 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:07.248488 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.250309 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.250312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:07.251402 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:07.251414 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.252383 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.252387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:07.252474 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.252498 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 10:44:07.252545 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 10:44:07.252610 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":1}
I20260927 10:44:07.252641 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.247350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44599 (local address 127.0.0.1:60238)
0927 10:44:07.248521 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:07.248536 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:07.248615 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:07.249032 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:07.249035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:07.249052 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:07.249142 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:07.249148 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.250063 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.250068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:07.251136 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:07.251146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.252239 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.252243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:07.252322 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.252345 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:44:07.252406 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 10:44:07.252479 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1113,"mutex_wait_us":45,"thread_start_us":94,"threads_started":1}
I20260927 10:44:07.253254 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.246975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56056 (local address 127.0.98.3:44633)
0927 10:44:07.247863 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:07.247874 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:07.247894 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:44:07.247958 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:07.247962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:07.248201 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:07.248376 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:07.248594 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.250180 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.251550 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.252244 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.252715 (+ 471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.252957 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:07.253002 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 10:44:07.253100 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":146,"threads_started":1}
I20260927 10:44:07.253610 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.247350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38249 (local address 127.0.0.1:52012)
0927 10:44:07.247600 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:07.247616 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:07.247704 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:07.249112 (+ 1408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:07.249115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:07.249134 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:07.249222 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:07.249227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.250716 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.250719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:07.251829 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:07.251839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.253263 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.253266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:07.253351 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.253373 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:44:07.253435 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 10:44:07.253494 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":1,"thread_start_us":101,"threads_started":1}
I20260927 10:44:07.253970 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.247876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52012 (local address 127.0.98.1:38249)
0927 10:44:07.248209 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:07.248220 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:07.248241 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:44:07.248389 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:07.248395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:07.248568 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:07.249379 (+ 811us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:07.249506 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.250598 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.251957 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.253142 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.253275 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.253555 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:07.253600 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 10:44:07.253817 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":107,"threads_started":1}
I20260927 10:44:07.255935 7661 tablet_service.cc:1467] Processing CreateTablet for tablet 9a804946082642ed92465f1f3e5270f4 (DEFAULT_TABLE table=test-table [id=2447ecc96f39498086b972c7691aaf4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:07.255962 7525 tablet_service.cc:1467] Processing CreateTablet for tablet 9a804946082642ed92465f1f3e5270f4 (DEFAULT_TABLE table=test-table [id=2447ecc96f39498086b972c7691aaf4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:07.256822 7661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a804946082642ed92465f1f3e5270f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:07.256524 7388 tablet_service.cc:1467] Processing CreateTablet for tablet 9a804946082642ed92465f1f3e5270f4 (DEFAULT_TABLE table=test-table [id=2447ecc96f39498086b972c7691aaf4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:07.256929 7525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a804946082642ed92465f1f3e5270f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:07.257376 7388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a804946082642ed92465f1f3e5270f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:07.264353 7762 tablet_bootstrap.cc:492] T 9a804946082642ed92465f1f3e5270f4 P ecaafcecaf3e414984d0c826a0c4246d: Bootstrap starting.
I20260927 10:44:07.264863 7761 tablet_bootstrap.cc:492] T 9a804946082642ed92465f1f3e5270f4 P babeb35fec414bac989abcc216118c74: Bootstrap starting.
I20260927 10:44:07.266733 7763 tablet_bootstrap.cc:492] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83: Bootstrap starting.
I20260927 10:44:07.267064 7761 tablet_bootstrap.cc:654] T 9a804946082642ed92465f1f3e5270f4 P babeb35fec414bac989abcc216118c74: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:07.267064 7762 tablet_bootstrap.cc:654] T 9a804946082642ed92465f1f3e5270f4 P ecaafcecaf3e414984d0c826a0c4246d: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:07.267984 7761 log.cc:824] T 9a804946082642ed92465f1f3e5270f4 P babeb35fec414bac989abcc216118c74: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:07.267984 7762 log.cc:824] T 9a804946082642ed92465f1f3e5270f4 P ecaafcecaf3e414984d0c826a0c4246d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:07.268646 7763 tablet_bootstrap.cc:654] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:07.269414 7763 log.cc:824] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:07.269992 7761 tablet_bootstrap.cc:492] T 9a804946082642ed92465f1f3e5270f4 P babeb35fec414bac989abcc216118c74: No bootstrap required, opened a new log
I20260927 10:44:07.270197 7761 ts_tablet_manager.cc:1397] T 9a804946082642ed92465f1f3e5270f4 P babeb35fec414bac989abcc216118c74: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:44:07.270510 7762 tablet_bootstrap.cc:492] T 9a804946082642ed92465f1f3e5270f4 P ecaafcecaf3e414984d0c826a0c4246d: No bootstrap required, opened a new log
I20260927 10:44:07.270681 7762 ts_tablet_manager.cc:1397] T 9a804946082642ed92465f1f3e5270f4 P ecaafcecaf3e414984d0c826a0c4246d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 10:44:07.271279 7763 tablet_bootstrap.cc:492] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83: No bootstrap required, opened a new log
I20260927 10:44:07.271461 7763 ts_tablet_manager.cc:1397] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:44:07.275233 7763 raft_consensus.cc:348] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babeb35fec414bac989abcc216118c74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44633 } } peers { permanent_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } } peers { permanent_uuid: "ecaafcecaf3e414984d0c826a0c4246d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44599 } }
I20260927 10:44:07.275317 7761 raft_consensus.cc:348] T 9a804946082642ed92465f1f3e5270f4 P babeb35fec414bac989abcc216118c74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babeb35fec414bac989abcc216118c74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44633 } } peers { permanent_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } } peers { permanent_uuid: "ecaafcecaf3e414984d0c826a0c4246d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44599 } }
I20260927 10:44:07.275537 7763 raft_consensus.cc:374] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:07.275554 7761 raft_consensus.cc:374] T 9a804946082642ed92465f1f3e5270f4 P babeb35fec414bac989abcc216118c74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:07.275591 7763 raft_consensus.cc:729] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c1e1840c784fdd871dc6b3572c1e83, State: Initialized, Role: FOLLOWER
I20260927 10:44:07.275606 7761 raft_consensus.cc:729] T 9a804946082642ed92465f1f3e5270f4 P babeb35fec414bac989abcc216118c74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: babeb35fec414bac989abcc216118c74, State: Initialized, Role: FOLLOWER
I20260927 10:44:07.275511 7762 raft_consensus.cc:348] T 9a804946082642ed92465f1f3e5270f4 P ecaafcecaf3e414984d0c826a0c4246d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babeb35fec414bac989abcc216118c74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44633 } } peers { permanent_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } } peers { permanent_uuid: "ecaafcecaf3e414984d0c826a0c4246d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44599 } }
I20260927 10:44:07.275728 7762 raft_consensus.cc:374] T 9a804946082642ed92465f1f3e5270f4 P ecaafcecaf3e414984d0c826a0c4246d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:07.275795 7762 raft_consensus.cc:729] T 9a804946082642ed92465f1f3e5270f4 P ecaafcecaf3e414984d0c826a0c4246d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecaafcecaf3e414984d0c826a0c4246d, State: Initialized, Role: FOLLOWER
I20260927 10:44:07.276062 7763 consensus_queue.cc:260] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babeb35fec414bac989abcc216118c74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44633 } } peers { permanent_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } } peers { permanent_uuid: "ecaafcecaf3e414984d0c826a0c4246d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44599 } }
I20260927 10:44:07.276105 7761 consensus_queue.cc:260] T 9a804946082642ed92465f1f3e5270f4 P babeb35fec414bac989abcc216118c74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babeb35fec414bac989abcc216118c74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44633 } } peers { permanent_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } } peers { permanent_uuid: "ecaafcecaf3e414984d0c826a0c4246d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44599 } }
I20260927 10:44:07.276129 7762 consensus_queue.cc:260] T 9a804946082642ed92465f1f3e5270f4 P ecaafcecaf3e414984d0c826a0c4246d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babeb35fec414bac989abcc216118c74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44633 } } peers { permanent_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } } peers { permanent_uuid: "ecaafcecaf3e414984d0c826a0c4246d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44599 } }
I20260927 10:44:07.277062 7762 ts_tablet_manager.cc:1428] T 9a804946082642ed92465f1f3e5270f4 P ecaafcecaf3e414984d0c826a0c4246d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:44:07.277071 7763 ts_tablet_manager.cc:1428] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:44:07.277976 7761 ts_tablet_manager.cc:1428] T 9a804946082642ed92465f1f3e5270f4 P babeb35fec414bac989abcc216118c74: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260927 10:44:07.278038 7728 heartbeater.cc:503] Master 127.0.98.62:34339 was elected leader, sending a full tablet report...
I20260927 10:44:07.278600 7455 heartbeater.cc:503] Master 127.0.98.62:34339 was elected leader, sending a full tablet report...
I20260927 10:44:07.278973 7592 heartbeater.cc:503] Master 127.0.98.62:34339 was elected leader, sending a full tablet report...
W20260927 10:44:07.296988 7457 tablet.cc:2369] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:44:07.394495 7767 raft_consensus.cc:482] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:44:07.394691 7767 raft_consensus.cc:504] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babeb35fec414bac989abcc216118c74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44633 } } peers { permanent_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } } peers { permanent_uuid: "ecaafcecaf3e414984d0c826a0c4246d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44599 } }
I20260927 10:44:07.395722 7767 leader_election.cc:288] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers babeb35fec414bac989abcc216118c74 (127.0.98.3:44633), ecaafcecaf3e414984d0c826a0c4246d (127.0.98.2:44599)
I20260927 10:44:07.399441 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.396109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47095 (local address 127.0.98.3:44633)
0927 10:44:07.396260 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:07.396264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:07.396382 (+ 118us) server_negotiation.cc:407] Connection header received
0927 10:44:07.396516 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:07.396520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:07.396570 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:07.396653 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:07.396921 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.397429 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.398516 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.398913 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.399031 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.399095 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:07.399256 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 10:44:07.399321 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:44:07.399462 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.396004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44633 (local address 127.0.98.1:47095)
0927 10:44:07.396268 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:07.396286 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:07.396398 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:07.396668 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:07.396671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:07.396688 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:07.396812 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:07.396818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.397538 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.397543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:07.398354 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:07.398362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.399036 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.399039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:07.399124 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.399149 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:44:07.399240 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 10:44:07.399310 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
I20260927 10:44:07.399658 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.396119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53431 (local address 127.0.98.2:44599)
0927 10:44:07.396268 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:07.396274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:07.396623 (+ 349us) server_negotiation.cc:407] Connection header received
0927 10:44:07.396775 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:07.396778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:07.396827 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:07.396910 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:07.397151 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.397642 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.398746 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.399140 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.399267 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.399323 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:07.399485 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 10:44:07.399546 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:44:07.399674 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.396004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44599 (local address 127.0.98.1:53431)
0927 10:44:07.396525 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:07.396542 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:07.396658 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:07.396918 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:07.396922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:07.396936 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:07.397042 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:07.397047 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.397757 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.397763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:07.398621 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:07.398630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.399267 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.399270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:07.399356 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.399384 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 10:44:07.399471 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 10:44:07.399538 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":291,"thread_start_us":122,"threads_started":1}
I20260927 10:44:07.400636 7681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a804946082642ed92465f1f3e5270f4" candidate_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "babeb35fec414bac989abcc216118c74" is_pre_election: true
I20260927 10:44:07.400681 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a804946082642ed92465f1f3e5270f4" candidate_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecaafcecaf3e414984d0c826a0c4246d" is_pre_election: true
I20260927 10:44:07.400946 7545 raft_consensus.cc:2454] T 9a804946082642ed92465f1f3e5270f4 P ecaafcecaf3e414984d0c826a0c4246d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5c1e1840c784fdd871dc6b3572c1e83 in term 0.
I20260927 10:44:07.400954 7681 raft_consensus.cc:2454] T 9a804946082642ed92465f1f3e5270f4 P babeb35fec414bac989abcc216118c74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5c1e1840c784fdd871dc6b3572c1e83 in term 0.
I20260927 10:44:07.401412 7343 leader_election.cc:302] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: babeb35fec414bac989abcc216118c74, e5c1e1840c784fdd871dc6b3572c1e83, ecaafcecaf3e414984d0c826a0c4246d; no voters:
I20260927 10:44:07.401674 7767 raft_consensus.cc:2781] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:44:07.401753 7767 raft_consensus.cc:482] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:44:07.401796 7767 raft_consensus.cc:3037] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:07.402822 7767 raft_consensus.cc:504] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babeb35fec414bac989abcc216118c74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44633 } } peers { permanent_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } } peers { permanent_uuid: "ecaafcecaf3e414984d0c826a0c4246d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44599 } }
I20260927 10:44:07.403147 7767 leader_election.cc:288] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83 [CANDIDATE]: Term 1 election: Requested vote from peers babeb35fec414bac989abcc216118c74 (127.0.98.3:44633), ecaafcecaf3e414984d0c826a0c4246d (127.0.98.2:44599)
I20260927 10:44:07.403498 7681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a804946082642ed92465f1f3e5270f4" candidate_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "babeb35fec414bac989abcc216118c74"
I20260927 10:44:07.403504 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a804946082642ed92465f1f3e5270f4" candidate_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecaafcecaf3e414984d0c826a0c4246d"
I20260927 10:44:07.403635 7681 raft_consensus.cc:3037] T 9a804946082642ed92465f1f3e5270f4 P babeb35fec414bac989abcc216118c74 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:07.403648 7545 raft_consensus.cc:3037] T 9a804946082642ed92465f1f3e5270f4 P ecaafcecaf3e414984d0c826a0c4246d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:07.404706 7545 raft_consensus.cc:2454] T 9a804946082642ed92465f1f3e5270f4 P ecaafcecaf3e414984d0c826a0c4246d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c1e1840c784fdd871dc6b3572c1e83 in term 1.
I20260927 10:44:07.404835 7681 raft_consensus.cc:2454] T 9a804946082642ed92465f1f3e5270f4 P babeb35fec414bac989abcc216118c74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c1e1840c784fdd871dc6b3572c1e83 in term 1.
I20260927 10:44:07.404989 7343 leader_election.cc:302] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e5c1e1840c784fdd871dc6b3572c1e83, ecaafcecaf3e414984d0c826a0c4246d; no voters:
I20260927 10:44:07.405249 7767 raft_consensus.cc:2781] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:07.405391 7767 raft_consensus.cc:686] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83 [term 1 LEADER]: Becoming Leader. State: Replica: e5c1e1840c784fdd871dc6b3572c1e83, State: Running, Role: LEADER
I20260927 10:44:07.405797 7767 consensus_queue.cc:237] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babeb35fec414bac989abcc216118c74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44633 } } peers { permanent_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } } peers { permanent_uuid: "ecaafcecaf3e414984d0c826a0c4246d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44599 } }
I20260927 10:44:07.409291 6489 catalog_manager.cc:5495] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83 reported cstate change: term changed from 0 to 1, leader changed from <none> to e5c1e1840c784fdd871dc6b3572c1e83 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babeb35fec414bac989abcc216118c74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecaafcecaf3e414984d0c826a0c4246d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44599 } health_report { overall_health: UNKNOWN } } }
I20260927 10:44:07.438844 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.435377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52018 (local address 127.0.98.1:38249)
0927 10:44:07.435514 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:07.435521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:07.435537 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:44:07.435621 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:07.435623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:07.435676 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:07.435787 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:44:07.436020 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.436527 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.437361 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.437597 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.437693 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.437986 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:44:07.438406 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:44:07.438593 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:07.438634 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 10:44:07.438689 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:44:07.444404 7681 raft_consensus.cc:1261] T 9a804946082642ed92465f1f3e5270f4 P babeb35fec414bac989abcc216118c74 [term 1 FOLLOWER]: Refusing update from remote peer e5c1e1840c784fdd871dc6b3572c1e83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:07.444404 7545 raft_consensus.cc:1261] T 9a804946082642ed92465f1f3e5270f4 P ecaafcecaf3e414984d0c826a0c4246d [term 1 FOLLOWER]: Refusing update from remote peer e5c1e1840c784fdd871dc6b3572c1e83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:07.445132 7772 consensus_queue.cc:1035] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "babeb35fec414bac989abcc216118c74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:07.445320 7767 consensus_queue.cc:1035] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecaafcecaf3e414984d0c826a0c4246d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 10:44:07.450591 7730 tablet.cc:2369] T 9a804946082642ed92465f1f3e5270f4 P babeb35fec414bac989abcc216118c74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:44:07.462556 6489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:07.462729 6489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:07.466765 6489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed2c254b7be546a19bad6932d621df69: Sending DeleteTablet for 3 replicas of tablet 9a804946082642ed92465f1f3e5270f4
I20260927 10:44:07.467490 7661 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a804946082642ed92465f1f3e5270f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56056
I20260927 10:44:07.467599 7525 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a804946082642ed92465f1f3e5270f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60238
I20260927 10:44:07.468051 7785 tablet_replica.cc:330] T 9a804946082642ed92465f1f3e5270f4 P babeb35fec414bac989abcc216118c74: stopping tablet replica
I20260927 10:44:07.468509 7785 raft_consensus.cc:2229] T 9a804946082642ed92465f1f3e5270f4 P babeb35fec414bac989abcc216118c74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:07.468555 7786 tablet_replica.cc:330] T 9a804946082642ed92465f1f3e5270f4 P ecaafcecaf3e414984d0c826a0c4246d: stopping tablet replica
I20260927 10:44:07.468778 7388 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a804946082642ed92465f1f3e5270f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52012
I20260927 10:44:07.468935 7786 raft_consensus.cc:2229] T 9a804946082642ed92465f1f3e5270f4 P ecaafcecaf3e414984d0c826a0c4246d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:07.469107 7785 raft_consensus.cc:2258] T 9a804946082642ed92465f1f3e5270f4 P babeb35fec414bac989abcc216118c74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:07.469221 7786 raft_consensus.cc:2258] T 9a804946082642ed92465f1f3e5270f4 P ecaafcecaf3e414984d0c826a0c4246d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:07.470510 7785 ts_tablet_manager.cc:1905] T 9a804946082642ed92465f1f3e5270f4 P babeb35fec414bac989abcc216118c74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:07.470999 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 6459
I20260927 10:44:07.471773 7786 ts_tablet_manager.cc:1905] T 9a804946082642ed92465f1f3e5270f4 P ecaafcecaf3e414984d0c826a0c4246d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:07.474131 7785 ts_tablet_manager.cc:1918] T 9a804946082642ed92465f1f3e5270f4 P babeb35fec414bac989abcc216118c74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:44:07.474253 7785 log.cc:1196] T 9a804946082642ed92465f1f3e5270f4 P babeb35fec414bac989abcc216118c74: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/9a804946082642ed92465f1f3e5270f4
I20260927 10:44:07.474406 7786 ts_tablet_manager.cc:1918] T 9a804946082642ed92465f1f3e5270f4 P ecaafcecaf3e414984d0c826a0c4246d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:44:07.474546 7786 log.cc:1196] T 9a804946082642ed92465f1f3e5270f4 P ecaafcecaf3e414984d0c826a0c4246d: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/wal/wals/9a804946082642ed92465f1f3e5270f4
I20260927 10:44:07.474664 7785 ts_tablet_manager.cc:1939] T 9a804946082642ed92465f1f3e5270f4 P babeb35fec414bac989abcc216118c74: Deleting consensus metadata
I20260927 10:44:07.474886 7786 ts_tablet_manager.cc:1939] T 9a804946082642ed92465f1f3e5270f4 P ecaafcecaf3e414984d0c826a0c4246d: Deleting consensus metadata
W20260927 10:44:07.478170 7615 connection.cc:535] server connection from 127.0.0.1:56056 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 10:44:07.478178 7480 connection.cc:535] server connection from 127.0.0.1:60238 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 10:44:07.485190 7787 tablet_replica.cc:330] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83: stopping tablet replica
I20260927 10:44:07.485574 7787 raft_consensus.cc:2229] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:44:07.486001 7787 raft_consensus.cc:2258] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:07.488023 7787 ts_tablet_manager.cc:1905] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:07.490470 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.469803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52558 (local address 127.0.98.61:44553)
0927 10:44:07.469921 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:07.469926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:07.475255 (+ 5329us) server_negotiation.cc:407] Connection header received
0927 10:44:07.475380 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:07.475386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:07.475458 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:07.475568 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:07.476863 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.477480 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.478840 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.479178 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.479281 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.481122 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:07.481128 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:07.481131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:07.481161 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:07.483045 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.485921 (+ 2876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.485924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.485925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.486009 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.486943 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.486946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.486948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.487189 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:07.487324 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:07.490121 (+ 2797us) server_negotiation.cc:299] Negotiation successful
0927 10:44:07.490275 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 10:44:07.491146 7787 ts_tablet_manager.cc:1918] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:44:07.491253 7787 log.cc:1196] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-0/wal/wals/9a804946082642ed92465f1f3e5270f4
I20260927 10:44:07.491595 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.481926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52564 (local address 127.0.98.61:44553)
0927 10:44:07.482026 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:07.482030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:07.482198 (+ 168us) server_negotiation.cc:407] Connection header received
0927 10:44:07.482371 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:07.482377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:07.482582 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:07.482642 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:07.483508 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.484303 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.485430 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.485732 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.485827 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.487790 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:07.487794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:07.487797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:07.487822 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:07.489503 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.490011 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.490015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.490017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.490104 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.490390 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.490393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.490395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.490616 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:07.490749 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:07.491346 (+ 597us) server_negotiation.cc:299] Negotiation successful
0927 10:44:07.491455 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 10:44:07.491601 7787 ts_tablet_manager.cc:1939] T 9a804946082642ed92465f1f3e5270f4 P e5c1e1840c784fdd871dc6b3572c1e83: Deleting consensus metadata
W20260927 10:44:07.492739 7342 connection.cc:441] server connection from 127.0.0.1:52012 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52012 (request call id 1) could send its response
I20260927 10:44:07.498747 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.469573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50100 (local address 127.0.98.60:44843)
0927 10:44:07.469927 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:07.469931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:07.488000 (+ 18069us) server_negotiation.cc:407] Connection header received
0927 10:44:07.488049 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:07.488052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:07.488212 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:07.488299 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:07.489226 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.489932 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.492936 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.493299 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.493394 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.494954 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:07.494958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:07.494960 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:07.494987 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:07.496879 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.497369 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.497372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.497374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.497456 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.497770 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.497779 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.497780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.498007 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:07.498142 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:07.498380 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 10:44:07.498549 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275}
I20260927 10:44:07.498754 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:07.482353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50104 (local address 127.0.98.60:44843)
0927 10:44:07.482536 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:07.482542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:07.483684 (+ 1142us) server_negotiation.cc:407] Connection header received
0927 10:44:07.483796 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:07.483799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:07.483854 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:07.483949 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:07.484524 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.485156 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.486376 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:07.486727 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:07.486826 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:07.495353 (+ 8527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:07.495360 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:07.495365 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:07.495394 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:07.497086 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.497536 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.497539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.497542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.497619 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:07.497909 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:07.497912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:07.497913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:07.498074 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:07.498167 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:07.498527 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 10:44:07.498619 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 10:44:08.458250 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:08.457638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.3:36573)
0927 10:44:08.457908 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:08.457994 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":73,"threads_started":1}
W20260927 10:44:08.458714 7728 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34339 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
I20260927 10:44:08.461136 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:08.460708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.1:44285)
0927 10:44:08.460916 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:08.461020 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":67,"threads_started":1}
W20260927 10:44:08.461710 7455 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34339 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
I20260927 10:44:08.462661 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:08.462192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.2:56307)
0927 10:44:08.462489 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:08.462574 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":56,"threads_started":1}
W20260927 10:44:08.462916 7592 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34339 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
I20260927 10:44:08.967490 7798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed2c254b7be546a19bad6932d621df69)
I20260927 10:44:08.967509 7799 raft_consensus.cc:482] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed2c254b7be546a19bad6932d621df69)
I20260927 10:44:08.967756 7799 raft_consensus.cc:504] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: false } }
I20260927 10:44:08.967756 7798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: false } }
I20260927 10:44:08.968780 7799 leader_election.cc:288] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed2c254b7be546a19bad6932d621df69 (127.0.98.62:34339), 258937809ebb4a2098f5a0f9d45a9ecb (127.0.98.61:44553)
I20260927 10:44:08.968788 7798 leader_election.cc:288] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed2c254b7be546a19bad6932d621df69 (127.0.98.62:34339), fec063b8e3804a5588d34385291a218b (127.0.98.60:44843)
I20260927 10:44:08.969339 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:08.968915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.0.1:37154)
0927 10:44:08.969152 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:08.969232 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
W20260927 10:44:08.969605 7262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
I20260927 10:44:08.969955 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:08.968941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.0.1:37162)
0927 10:44:08.969510 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:08.969850 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":60,"threads_started":1}
W20260927 10:44:08.970120 7078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
I20260927 10:44:08.970510 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:08.970250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.0.1:37170)
0927 10:44:08.970383 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:08.970447 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260927 10:44:08.970651 7262 leader_election.cc:334] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ed2c254b7be546a19bad6932d621df69 (127.0.98.62:34339): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
I20260927 10:44:08.971025 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:08.970778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.0.1:37178)
0927 10:44:08.970901 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:08.970967 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260927 10:44:08.971146 7078 leader_election.cc:334] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ed2c254b7be546a19bad6932d621df69 (127.0.98.62:34339): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
I20260927 10:44:08.973459 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:08.969248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44843 (local address 127.0.0.1:50114)
0927 10:44:08.969752 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:08.969764 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:08.969862 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:08.970305 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:08.970310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:08.970334 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:08.970475 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:08.970483 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:08.971516 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:08.971523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:08.972378 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:08.972386 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:08.973064 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:08.973067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:08.973162 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:08.973183 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:44:08.973269 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 10:44:08.973338 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":66,"threads_started":1}
I20260927 10:44:08.973469 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:08.969419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50114 (local address 127.0.98.60:44843)
0927 10:44:08.969730 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:08.969735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:08.969842 (+ 107us) server_negotiation.cc:407] Connection header received
0927 10:44:08.969982 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:08.969986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:08.970036 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:08.970173 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:08.970604 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:08.971407 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:08.972499 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:08.972952 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:08.973065 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:08.973123 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:08.973285 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 10:44:08.973349 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":48,"threads_started":1}
I20260927 10:44:08.973659 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:08.969707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52580 (local address 127.0.98.61:44553)
0927 10:44:08.969994 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:08.970114 (+ 120us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:08.970129 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:44:08.970178 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:08.970184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:08.970282 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:08.970370 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:08.970910 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:08.971571 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:08.972705 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:08.973118 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:08.973231 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:08.973284 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:08.973449 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 10:44:08.973515 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":75,"threads_started":1}
I20260927 10:44:08.973920 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:08.969240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44553 (local address 127.0.0.1:52580)
0927 10:44:08.969439 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:08.969453 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:08.969561 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:08.970502 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:08.970505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:08.970524 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:08.970693 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:08.970701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:08.971704 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:08.971710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:08.972566 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:08.972575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:08.973233 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:08.973237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:08.973330 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:08.973347 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 10:44:08.973434 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 10:44:08.973814 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":62,"threads_started":1}
I20260927 10:44:08.973943 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fec063b8e3804a5588d34385291a218b" is_pre_election: true
I20260927 10:44:08.974171 7285 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 258937809ebb4a2098f5a0f9d45a9ecb in term 1.
I20260927 10:44:08.974439 7100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fec063b8e3804a5588d34385291a218b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" is_pre_election: true
I20260927 10:44:08.974591 7078 leader_election.cc:302] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 258937809ebb4a2098f5a0f9d45a9ecb, fec063b8e3804a5588d34385291a218b; no voters: ed2c254b7be546a19bad6932d621df69
I20260927 10:44:08.974622 7100 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec063b8e3804a5588d34385291a218b in term 1.
I20260927 10:44:08.974822 7798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 10:44:08.974875 7798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 1 FOLLOWER]: Starting leader election (detected failure of leader ed2c254b7be546a19bad6932d621df69)
I20260927 10:44:08.974915 7798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:44:08.974982 7262 leader_election.cc:302] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 258937809ebb4a2098f5a0f9d45a9ecb, fec063b8e3804a5588d34385291a218b; no voters: ed2c254b7be546a19bad6932d621df69
I20260927 10:44:08.975164 7799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 10:44:08.975214 7799 raft_consensus.cc:482] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [term 1 FOLLOWER]: Starting leader election (detected failure of leader ed2c254b7be546a19bad6932d621df69)
I20260927 10:44:08.975251 7799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:44:08.976076 7798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: false } }
I20260927 10:44:08.976285 7799 raft_consensus.cc:504] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: false } }
I20260927 10:44:08.976438 7798 leader_election.cc:288] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [CANDIDATE]: Term 2 election: Requested vote from peers ed2c254b7be546a19bad6932d621df69 (127.0.98.62:34339), fec063b8e3804a5588d34385291a218b (127.0.98.60:44843)
I20260927 10:44:08.976594 7799 leader_election.cc:288] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [CANDIDATE]: Term 2 election: Requested vote from peers ed2c254b7be546a19bad6932d621df69 (127.0.98.62:34339), 258937809ebb4a2098f5a0f9d45a9ecb (127.0.98.61:44553)
I20260927 10:44:08.976745 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:08.976537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.0.1:37186)
0927 10:44:08.976622 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:08.976677 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 10:44:08.977005 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fec063b8e3804a5588d34385291a218b"
I20260927 10:44:08.977111 7100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fec063b8e3804a5588d34385291a218b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "258937809ebb4a2098f5a0f9d45a9ecb"
I20260927 10:44:08.977185 7285 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 258937809ebb4a2098f5a0f9d45a9ecb in current term 2: Already voted for candidate fec063b8e3804a5588d34385291a218b in this term.
I20260927 10:44:08.977237 7100 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fec063b8e3804a5588d34385291a218b in current term 2: Already voted for candidate 258937809ebb4a2098f5a0f9d45a9ecb in this term.
I20260927 10:44:08.977274 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:08.976681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.0.1:37196)
0927 10:44:08.977119 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:08.977169 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364}
I20260927 10:44:08.977514 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:08.977155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.0.1:37202)
0927 10:44:08.977401 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:08.977458 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
W20260927 10:44:08.977653 7078 leader_election.cc:334] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ed2c254b7be546a19bad6932d621df69 (127.0.98.62:34339): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
I20260927 10:44:08.977773 7078 leader_election.cc:302] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 258937809ebb4a2098f5a0f9d45a9ecb; no voters: ed2c254b7be546a19bad6932d621df69, fec063b8e3804a5588d34385291a218b
I20260927 10:44:08.977914 7798 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 10:44:08.978003 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:08.977794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.0.1:37212)
0927 10:44:08.977888 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:08.977942 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 10:44:08.978091 7262 leader_election.cc:334] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ed2c254b7be546a19bad6932d621df69 (127.0.98.62:34339): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
I20260927 10:44:08.978165 7262 leader_election.cc:302] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fec063b8e3804a5588d34385291a218b; no voters: 258937809ebb4a2098f5a0f9d45a9ecb, ed2c254b7be546a19bad6932d621df69
I20260927 10:44:08.978330 7799 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 10:44:09.460150 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:09.459521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.3:57855)
0927 10:44:09.459935 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:09.460034 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":102,"threads_started":1}
I20260927 10:44:09.462836 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:09.462281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.1:46965)
0927 10:44:09.462612 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:09.462673 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":66,"threads_started":1}
I20260927 10:44:09.463688 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:09.463333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.2:45979)
0927 10:44:09.463551 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:09.463616 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":92,"threads_started":1}
I20260927 10:44:10.461891 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:10.461333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.3:49539)
0927 10:44:10.461702 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:10.461776 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":104,"threads_started":1}
I20260927 10:44:10.464165 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:10.463766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.1:59329)
0927 10:44:10.463996 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:10.464066 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":71,"threads_started":1}
I20260927 10:44:10.464740 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:10.464402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.2:60161)
0927 10:44:10.464620 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:10.464668 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":83,"threads_started":1}
I20260927 10:44:10.917064 7814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:44:10.917212 7814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: false } }
I20260927 10:44:10.917693 7814 leader_election.cc:288] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ed2c254b7be546a19bad6932d621df69 (127.0.98.62:34339), fec063b8e3804a5588d34385291a218b (127.0.98.60:44843)
I20260927 10:44:10.918304 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:10.917856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.0.1:37466)
0927 10:44:10.918086 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:10.918202 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":70,"threads_started":1}
I20260927 10:44:10.918550 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fec063b8e3804a5588d34385291a218b" is_pre_election: true
I20260927 10:44:10.918723 7285 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 258937809ebb4a2098f5a0f9d45a9ecb in term 2.
I20260927 10:44:10.919124 7078 leader_election.cc:302] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 258937809ebb4a2098f5a0f9d45a9ecb, fec063b8e3804a5588d34385291a218b; no voters:
I20260927 10:44:10.919216 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:10.918988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.0.1:37476)
0927 10:44:10.919111 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:10.919162 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 10:44:10.919301 7814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260927 10:44:10.919478 7078 leader_election.cc:334] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ed2c254b7be546a19bad6932d621df69 (127.0.98.62:34339): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
I20260927 10:44:10.919548 7814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:44:10.919601 7814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 2 FOLLOWER]: Advancing to term 3
I20260927 10:44:10.920616 7814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: false } }
I20260927 10:44:10.920966 7814 leader_election.cc:288] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [CANDIDATE]: Term 3 election: Requested vote from peers ed2c254b7be546a19bad6932d621df69 (127.0.98.62:34339), fec063b8e3804a5588d34385291a218b (127.0.98.60:44843)
I20260927 10:44:10.921389 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:10.921138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.0.1:37490)
0927 10:44:10.921247 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:10.921324 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 10:44:10.921581 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fec063b8e3804a5588d34385291a218b"
I20260927 10:44:10.921723 7285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [term 2 FOLLOWER]: Advancing to term 3
I20260927 10:44:10.922755 7285 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 258937809ebb4a2098f5a0f9d45a9ecb in term 3.
I20260927 10:44:10.923054 7078 leader_election.cc:302] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 258937809ebb4a2098f5a0f9d45a9ecb, fec063b8e3804a5588d34385291a218b; no voters:
I20260927 10:44:10.923122 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:10.922916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.0.1:37492)
0927 10:44:10.923026 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:10.923069 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 10:44:10.923245 7078 leader_election.cc:334] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ed2c254b7be546a19bad6932d621df69 (127.0.98.62:34339): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
I20260927 10:44:10.923302 7814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 3 FOLLOWER]: Leader election won for term 3
I20260927 10:44:10.923481 7814 raft_consensus.cc:686] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [term 3 LEADER]: Becoming Leader. State: Replica: 258937809ebb4a2098f5a0f9d45a9ecb, State: Running, Role: LEADER
I20260927 10:44:10.923743 7814 consensus_queue.cc:237] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: false } }
I20260927 10:44:10.925113 7817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 258937809ebb4a2098f5a0f9d45a9ecb. Latest consensus state: current_term: 3 leader_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: false } } }
I20260927 10:44:10.925254 7817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:10.925593 7819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:44:10.927940 7819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:44:10.928334 7819 catalog_manager.cc:1259] Loaded cluster ID: 8d324f4216d949b7b8161f188793fea6
I20260927 10:44:10.928386 7819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:44:10.928946 7819 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:44:10.929330 7819 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb: Loaded TSK: 0
I20260927 10:44:10.930073 7819 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:44:10.968549 7090 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:44:10.971820 7090 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:44:10.975121 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:10.974829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.0.1:37508)
0927 10:44:10.974949 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:10.975013 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 10:44:10.975426 7285 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [term 3 FOLLOWER]: Refusing update from remote peer 258937809ebb4a2098f5a0f9d45a9ecb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 10:44:10.975813 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:10.975574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.0.1:37516)
0927 10:44:10.975666 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:10.975734 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 10:44:10.975922 7078 consensus_peers.cc:487] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb -> Peer ed2c254b7be546a19bad6932d621df69 (127.0.98.62:34339): Couldn't send request to peer ed2c254b7be546a19bad6932d621df69. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 10:44:10.975960 7814 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 10:44:10.977401 7821 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 258937809ebb4a2098f5a0f9d45a9ecb. Latest consensus state: current_term: 3 leader_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: false } } }
I20260927 10:44:10.977562 7821 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:44:10.978786 7817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: false } } }
I20260927 10:44:10.978915 7817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:10.979226 7821 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ed2c254b7be546a19bad6932d621df69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34339 } } peers { permanent_uuid: "258937809ebb4a2098f5a0f9d45a9ecb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attrs { promote: false } } peers { permanent_uuid: "fec063b8e3804a5588d34385291a218b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44843 } attrs { promote: false } } }
I20260927 10:44:10.979344 7821 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec063b8e3804a5588d34385291a218b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:44:10.992594 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:10.986231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38249 (local address 127.0.0.1:52024)
0927 10:44:10.986554 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:10.986572 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:10.986669 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:10.987602 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:10.987605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:10.987623 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:10.987732 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:10.987739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:10.988930 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:10.988947 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:10.990370 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:10.990415 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:10.992053 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:10.992060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:10.992185 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:10.992225 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 10:44:10.992300 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 10:44:10.992395 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":1}
I20260927 10:44:10.993180 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:10.986500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52024 (local address 127.0.98.1:38249)
0927 10:44:10.987100 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:10.987104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:10.987120 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:44:10.987176 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:10.987179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:10.987228 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:10.987314 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:10.987870 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:10.988650 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:10.990931 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:10.991607 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:10.991892 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:10.991987 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:10.992817 (+ 830us) server_negotiation.cc:299] Negotiation successful
0927 10:44:10.992901 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":87,"threads_started":1}
I20260927 10:44:10.994522 7388 tablet_service.cc:1467] Processing CreateTablet for tablet d9e021f87fd44055bf7cd647e22d13a8 (DEFAULT_TABLE table=test-table [id=71f2b3d11345453ea260c44707cbcd21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:10.994988 7388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e021f87fd44055bf7cd647e22d13a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:10.995066 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:10.988369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44633 (local address 127.0.0.1:56072)
0927 10:44:10.990694 (+ 2325us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:10.990709 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:10.990811 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:10.991231 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:10.991234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:10.991248 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:10.991357 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:10.991364 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:10.992493 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:10.992496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:10.993572 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:10.993583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:10.994666 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:10.994670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:10.994767 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:10.994794 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:44:10.994861 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 10:44:10.994940 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2241,"thread_start_us":89,"threads_started":1}
I20260927 10:44:10.995101 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:10.986821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60242 (local address 127.0.98.2:44599)
0927 10:44:10.987115 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:10.987121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:10.988763 (+ 1642us) server_negotiation.cc:407] Connection header received
0927 10:44:10.988951 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:10.988954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:10.988999 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:10.989078 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:10.989436 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:10.989948 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:10.992373 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:10.993031 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:10.993115 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:10.993174 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:10.993469 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 10:44:10.994973 (+ 1504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
I20260927 10:44:10.995476 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:10.986657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44599 (local address 127.0.0.1:60242)
0927 10:44:10.988676 (+ 2019us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:10.988692 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:10.988827 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:10.989190 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:10.989193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:10.989210 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:10.989326 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:10.989335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:10.990061 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:10.990067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:10.992237 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:10.992247 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:10.993258 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:10.993261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:10.993343 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:10.993362 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 10:44:10.995280 (+ 1918us) client_negotiation.cc:240] Negotiation successful
0927 10:44:10.995354 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1959,"mutex_wait_us":3,"thread_start_us":96,"threads_started":1}
I20260927 10:44:10.995476 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:10.986022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56072 (local address 127.0.98.3:44633)
0927 10:44:10.987892 (+ 1870us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:10.987896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:10.990783 (+ 2887us) server_negotiation.cc:407] Connection header received
0927 10:44:10.990908 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:10.990911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:10.990993 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:10.991106 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:10.991487 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:10.992366 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:10.993707 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:10.994366 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:10.994564 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:10.994623 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:10.995282 (+ 659us) server_negotiation.cc:299] Negotiation successful
0927 10:44:10.995356 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1787,"thread_start_us":115,"threads_started":1}
I20260927 10:44:10.996111 7525 tablet_service.cc:1467] Processing CreateTablet for tablet d9e021f87fd44055bf7cd647e22d13a8 (DEFAULT_TABLE table=test-table [id=71f2b3d11345453ea260c44707cbcd21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:10.996435 7525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e021f87fd44055bf7cd647e22d13a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:10.997946 7831 tablet_bootstrap.cc:492] T d9e021f87fd44055bf7cd647e22d13a8 P e5c1e1840c784fdd871dc6b3572c1e83: Bootstrap starting.
I20260927 10:44:10.998533 7661 tablet_service.cc:1467] Processing CreateTablet for tablet d9e021f87fd44055bf7cd647e22d13a8 (DEFAULT_TABLE table=test-table [id=71f2b3d11345453ea260c44707cbcd21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:10.998885 7661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e021f87fd44055bf7cd647e22d13a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:10.999382 7831 tablet_bootstrap.cc:654] T d9e021f87fd44055bf7cd647e22d13a8 P e5c1e1840c784fdd871dc6b3572c1e83: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:10.999637 7832 tablet_bootstrap.cc:492] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d: Bootstrap starting.
I20260927 10:44:11.000507 7832 tablet_bootstrap.cc:654] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:11.000515 7831 tablet_bootstrap.cc:492] T d9e021f87fd44055bf7cd647e22d13a8 P e5c1e1840c784fdd871dc6b3572c1e83: No bootstrap required, opened a new log
I20260927 10:44:11.000593 7831 ts_tablet_manager.cc:1397] T d9e021f87fd44055bf7cd647e22d13a8 P e5c1e1840c784fdd871dc6b3572c1e83: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:44:11.000994 7831 raft_consensus.cc:348] T d9e021f87fd44055bf7cd647e22d13a8 P e5c1e1840c784fdd871dc6b3572c1e83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babeb35fec414bac989abcc216118c74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44633 } } peers { permanent_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } } peers { permanent_uuid: "ecaafcecaf3e414984d0c826a0c4246d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44599 } }
I20260927 10:44:11.001108 7831 raft_consensus.cc:374] T d9e021f87fd44055bf7cd647e22d13a8 P e5c1e1840c784fdd871dc6b3572c1e83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:11.001140 7831 raft_consensus.cc:729] T d9e021f87fd44055bf7cd647e22d13a8 P e5c1e1840c784fdd871dc6b3572c1e83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c1e1840c784fdd871dc6b3572c1e83, State: Initialized, Role: FOLLOWER
I20260927 10:44:11.001289 7831 consensus_queue.cc:260] T d9e021f87fd44055bf7cd647e22d13a8 P e5c1e1840c784fdd871dc6b3572c1e83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babeb35fec414bac989abcc216118c74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44633 } } peers { permanent_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } } peers { permanent_uuid: "ecaafcecaf3e414984d0c826a0c4246d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44599 } }
I20260927 10:44:11.001608 7831 ts_tablet_manager.cc:1428] T d9e021f87fd44055bf7cd647e22d13a8 P e5c1e1840c784fdd871dc6b3572c1e83: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:44:11.001744 7835 tablet_bootstrap.cc:492] T d9e021f87fd44055bf7cd647e22d13a8 P babeb35fec414bac989abcc216118c74: Bootstrap starting.
I20260927 10:44:11.002818 7835 tablet_bootstrap.cc:654] T d9e021f87fd44055bf7cd647e22d13a8 P babeb35fec414bac989abcc216118c74: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:11.004402 7835 tablet_bootstrap.cc:492] T d9e021f87fd44055bf7cd647e22d13a8 P babeb35fec414bac989abcc216118c74: No bootstrap required, opened a new log
I20260927 10:44:11.004562 7835 ts_tablet_manager.cc:1397] T d9e021f87fd44055bf7cd647e22d13a8 P babeb35fec414bac989abcc216118c74: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:44:11.004822 7832 tablet_bootstrap.cc:492] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d: No bootstrap required, opened a new log
I20260927 10:44:11.004982 7832 ts_tablet_manager.cc:1397] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 10:44:11.005200 7835 raft_consensus.cc:348] T d9e021f87fd44055bf7cd647e22d13a8 P babeb35fec414bac989abcc216118c74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babeb35fec414bac989abcc216118c74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44633 } } peers { permanent_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } } peers { permanent_uuid: "ecaafcecaf3e414984d0c826a0c4246d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44599 } }
I20260927 10:44:11.005311 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:11.002387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.1:41323)
0927 10:44:11.005120 (+ 2733us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:11.005193 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2653,"thread_start_us":460,"threads_started":1}
I20260927 10:44:11.005323 7835 raft_consensus.cc:374] T d9e021f87fd44055bf7cd647e22d13a8 P babeb35fec414bac989abcc216118c74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:11.005362 7835 raft_consensus.cc:729] T d9e021f87fd44055bf7cd647e22d13a8 P babeb35fec414bac989abcc216118c74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: babeb35fec414bac989abcc216118c74, State: Initialized, Role: FOLLOWER
I20260927 10:44:11.005527 7835 consensus_queue.cc:260] T d9e021f87fd44055bf7cd647e22d13a8 P babeb35fec414bac989abcc216118c74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babeb35fec414bac989abcc216118c74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44633 } } peers { permanent_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } } peers { permanent_uuid: "ecaafcecaf3e414984d0c826a0c4246d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44599 } }
I20260927 10:44:11.005616 7832 raft_consensus.cc:348] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babeb35fec414bac989abcc216118c74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44633 } } peers { permanent_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } } peers { permanent_uuid: "ecaafcecaf3e414984d0c826a0c4246d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44599 } }
I20260927 10:44:11.005806 7832 raft_consensus.cc:374] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:11.005860 7832 raft_consensus.cc:729] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecaafcecaf3e414984d0c826a0c4246d, State: Initialized, Role: FOLLOWER
I20260927 10:44:11.005942 7835 ts_tablet_manager.cc:1428] T d9e021f87fd44055bf7cd647e22d13a8 P babeb35fec414bac989abcc216118c74: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:44:11.006014 7832 consensus_queue.cc:260] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babeb35fec414bac989abcc216118c74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44633 } } peers { permanent_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } } peers { permanent_uuid: "ecaafcecaf3e414984d0c826a0c4246d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44599 } }
I20260927 10:44:11.007082 7832 ts_tablet_manager.cc:1428] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:44:11.008780 7729 heartbeater.cc:503] Master 127.0.98.61:44553 was elected leader, sending a full tablet report...
I20260927 10:44:11.008883 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:11.008467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.2:44833)
0927 10:44:11.008721 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:11.008779 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":91,"threads_started":1}
W20260927 10:44:11.011749 7594 tablet.cc:2369] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:44:11.014816 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:11.006744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.3:54587)
0927 10:44:11.014618 (+ 7874us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:11.014691 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7784,"thread_start_us":84,"threads_started":1}
I20260927 10:44:11.015578 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:11.015342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.3:44627)
0927 10:44:11.015465 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:11.015519 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 10:44:11.378285 7840 raft_consensus.cc:482] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:44:11.378501 7840 raft_consensus.cc:504] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babeb35fec414bac989abcc216118c74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44633 } } peers { permanent_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } } peers { permanent_uuid: "ecaafcecaf3e414984d0c826a0c4246d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44599 } }
I20260927 10:44:11.379657 7840 leader_election.cc:288] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers babeb35fec414bac989abcc216118c74 (127.0.98.3:44633), e5c1e1840c784fdd871dc6b3572c1e83 (127.0.98.1:38249)
I20260927 10:44:11.384436 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:11.380019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43609 (local address 127.0.98.3:44633)
0927 10:44:11.380180 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:11.380185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:11.380202 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:44:11.380384 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:11.380388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:11.380439 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:11.380546 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:11.380891 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:11.381687 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:11.383408 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:11.383834 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:11.383918 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:11.384010 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:11.384217 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 10:44:11.384298 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 10:44:11.384470 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:11.380093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40569 (local address 127.0.98.1:38249)
0927 10:44:11.380233 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:11.380238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:11.380601 (+ 363us) server_negotiation.cc:407] Connection header received
0927 10:44:11.380723 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:11.380726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:11.380791 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:11.380881 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:11.381071 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:11.381772 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:11.383446 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:11.383866 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:11.383998 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:11.384062 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:11.384281 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 10:44:11.384367 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:44:11.384447 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:11.379944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44633 (local address 127.0.98.2:43609)
0927 10:44:11.380106 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:11.380124 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:11.380238 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:11.380587 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:11.380591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:11.380626 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:11.380772 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:11.380778 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:11.381857 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:11.381862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:11.383254 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:11.383264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:11.383970 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:11.383974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:11.384073 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:11.384102 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 10:44:11.384174 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 10:44:11.384269 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":112,"spinlock_wait_cycles":3200}
I20260927 10:44:11.384469 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:11.379955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38249 (local address 127.0.98.2:40569)
0927 10:44:11.380487 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:11.380499 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:11.380632 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:11.380901 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:11.380904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:11.380926 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:11.380983 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:11.380987 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:11.381888 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:11.381892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:11.383266 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:11.383278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:11.384013 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:11.384018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:11.384122 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:11.384147 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:44:11.384245 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 10:44:11.384312 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":277,"spinlock_wait_cycles":20352,"thread_start_us":158,"threads_started":1}
I20260927 10:44:11.385166 7681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e021f87fd44055bf7cd647e22d13a8" candidate_uuid: "ecaafcecaf3e414984d0c826a0c4246d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "babeb35fec414bac989abcc216118c74" is_pre_election: true
I20260927 10:44:11.385166 7408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e021f87fd44055bf7cd647e22d13a8" candidate_uuid: "ecaafcecaf3e414984d0c826a0c4246d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" is_pre_election: true
I20260927 10:44:11.385335 7681 raft_consensus.cc:2454] T d9e021f87fd44055bf7cd647e22d13a8 P babeb35fec414bac989abcc216118c74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecaafcecaf3e414984d0c826a0c4246d in term 0.
I20260927 10:44:11.385393 7408 raft_consensus.cc:2454] T d9e021f87fd44055bf7cd647e22d13a8 P e5c1e1840c784fdd871dc6b3572c1e83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecaafcecaf3e414984d0c826a0c4246d in term 0.
I20260927 10:44:11.385771 7481 leader_election.cc:302] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: babeb35fec414bac989abcc216118c74, e5c1e1840c784fdd871dc6b3572c1e83, ecaafcecaf3e414984d0c826a0c4246d; no voters:
I20260927 10:44:11.386065 7840 raft_consensus.cc:2781] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:44:11.386128 7840 raft_consensus.cc:482] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:44:11.386165 7840 raft_consensus.cc:3037] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:11.387203 7840 raft_consensus.cc:504] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babeb35fec414bac989abcc216118c74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44633 } } peers { permanent_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } } peers { permanent_uuid: "ecaafcecaf3e414984d0c826a0c4246d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44599 } }
I20260927 10:44:11.387580 7840 leader_election.cc:288] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d [CANDIDATE]: Term 1 election: Requested vote from peers babeb35fec414bac989abcc216118c74 (127.0.98.3:44633), e5c1e1840c784fdd871dc6b3572c1e83 (127.0.98.1:38249)
I20260927 10:44:11.387943 7681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e021f87fd44055bf7cd647e22d13a8" candidate_uuid: "ecaafcecaf3e414984d0c826a0c4246d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "babeb35fec414bac989abcc216118c74"
I20260927 10:44:11.388096 7681 raft_consensus.cc:3037] T d9e021f87fd44055bf7cd647e22d13a8 P babeb35fec414bac989abcc216118c74 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:11.388058 7408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e021f87fd44055bf7cd647e22d13a8" candidate_uuid: "ecaafcecaf3e414984d0c826a0c4246d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c1e1840c784fdd871dc6b3572c1e83"
I20260927 10:44:11.388221 7408 raft_consensus.cc:3037] T d9e021f87fd44055bf7cd647e22d13a8 P e5c1e1840c784fdd871dc6b3572c1e83 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:11.389027 7681 raft_consensus.cc:2454] T d9e021f87fd44055bf7cd647e22d13a8 P babeb35fec414bac989abcc216118c74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecaafcecaf3e414984d0c826a0c4246d in term 1.
I20260927 10:44:11.389240 7408 raft_consensus.cc:2454] T d9e021f87fd44055bf7cd647e22d13a8 P e5c1e1840c784fdd871dc6b3572c1e83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecaafcecaf3e414984d0c826a0c4246d in term 1.
I20260927 10:44:11.389314 7481 leader_election.cc:302] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: babeb35fec414bac989abcc216118c74, ecaafcecaf3e414984d0c826a0c4246d; no voters:
I20260927 10:44:11.389496 7840 raft_consensus.cc:2781] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:11.389740 7840 raft_consensus.cc:686] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d [term 1 LEADER]: Becoming Leader. State: Replica: ecaafcecaf3e414984d0c826a0c4246d, State: Running, Role: LEADER
I20260927 10:44:11.390000 7840 consensus_queue.cc:237] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babeb35fec414bac989abcc216118c74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44633 } } peers { permanent_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } } peers { permanent_uuid: "ecaafcecaf3e414984d0c826a0c4246d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44599 } }
I20260927 10:44:11.390105 7593 heartbeater.cc:503] Master 127.0.98.61:44553 was elected leader, sending a full tablet report...
I20260927 10:44:11.390552 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:11.390204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.2:34861)
0927 10:44:11.390388 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:11.390473 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260927 10:44:11.392504 7089 catalog_manager.cc:5495] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d reported cstate change: term changed from 0 to 1, leader changed from <none> to ecaafcecaf3e414984d0c826a0c4246d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ecaafcecaf3e414984d0c826a0c4246d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babeb35fec414bac989abcc216118c74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecaafcecaf3e414984d0c826a0c4246d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44599 } health_report { overall_health: HEALTHY } } }
I20260927 10:44:11.449760 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:11.446356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60252 (local address 127.0.98.2:44599)
0927 10:44:11.446509 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:11.446512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:11.446544 (+ 32us) server_negotiation.cc:407] Connection header received
0927 10:44:11.446638 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:11.446641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:11.446716 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:11.446809 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:44:11.447067 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:11.447588 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:11.448404 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:11.448652 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:11.448750 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:11.448949 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:44:11.449431 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:44:11.449527 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:11.449594 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 10:44:11.449644 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:44:11.452730 7681 raft_consensus.cc:1261] T d9e021f87fd44055bf7cd647e22d13a8 P babeb35fec414bac989abcc216118c74 [term 1 FOLLOWER]: Refusing update from remote peer ecaafcecaf3e414984d0c826a0c4246d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:11.452958 7408 raft_consensus.cc:1261] T d9e021f87fd44055bf7cd647e22d13a8 P e5c1e1840c784fdd871dc6b3572c1e83 [term 1 FOLLOWER]: Refusing update from remote peer ecaafcecaf3e414984d0c826a0c4246d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:11.453265 7844 consensus_queue.cc:1035] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d [LEADER]: Connected to new peer: Peer: permanent_uuid: "babeb35fec414bac989abcc216118c74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:11.453553 7840 consensus_queue.cc:1035] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c1e1840c784fdd871dc6b3572c1e83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:11.455080 7456 heartbeater.cc:503] Master 127.0.98.61:44553 was elected leader, sending a full tablet report...
I20260927 10:44:11.455283 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:11.454885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.3:54091)
0927 10:44:11.455147 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:11.455206 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260927 10:44:11.455524 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:11.455212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.1:33399)
0927 10:44:11.455390 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:11.455452 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260927 10:44:11.457386 7848 mvcc.cc:205] Tried to move back new op lower bound from 7333911967544623104 to 7333911967296442368. Current Snapshot: MvccSnapshot[applied={T|T < 7333911967544623104}]
I20260927 10:44:11.458052 7851 mvcc.cc:205] Tried to move back new op lower bound from 7333911967544623104 to 7333911967296442368. Current Snapshot: MvccSnapshot[applied={T|T < 7333911967544623104}]
I20260927 10:44:11.458361 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:11.457971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.2:36599)
0927 10:44:11.458226 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:11.458286 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260927 10:44:11.460810 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:11.460596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.1:60719)
0927 10:44:11.460709 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:11.460754 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 10:44:11.461946 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:11.461493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.2:53247)
0927 10:44:11.461788 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:11.461857 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224}
I20260927 10:44:11.462195 7849 mvcc.cc:205] Tried to move back new op lower bound from 7333911967544623104 to 7333911967296442368. Current Snapshot: MvccSnapshot[applied={T|T < 7333911967544623104}]
I20260927 10:44:11.463017 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:11.462745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34339 (local address 127.0.98.3:42575)
0927 10:44:11.462897 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:11.462952 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 10:44:11.466988 7089 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:11.467160 7089 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:11.471426 7089 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 258937809ebb4a2098f5a0f9d45a9ecb: Sending DeleteTablet for 3 replicas of tablet d9e021f87fd44055bf7cd647e22d13a8
I20260927 10:44:11.472136 7388 tablet_service.cc:1514] Processing DeleteTablet for tablet d9e021f87fd44055bf7cd647e22d13a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:11 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:52024
I20260927 10:44:11.472357 7525 tablet_service.cc:1514] Processing DeleteTablet for tablet d9e021f87fd44055bf7cd647e22d13a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:11 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:60242
I20260927 10:44:11.473098 7858 tablet_replica.cc:330] T d9e021f87fd44055bf7cd647e22d13a8 P e5c1e1840c784fdd871dc6b3572c1e83: stopping tablet replica
I20260927 10:44:11.473268 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 7235
I20260927 10:44:11.473327 7661 tablet_service.cc:1514] Processing DeleteTablet for tablet d9e021f87fd44055bf7cd647e22d13a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:11 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:56072
I20260927 10:44:11.473892 7859 tablet_replica.cc:330] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d: stopping tablet replica
I20260927 10:44:11.474021 7859 raft_consensus.cc:2229] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:44:11.474362 7859 raft_consensus.cc:2258] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:11.475080 7860 tablet_replica.cc:330] T d9e021f87fd44055bf7cd647e22d13a8 P babeb35fec414bac989abcc216118c74: stopping tablet replica
I20260927 10:44:11.475208 7859 ts_tablet_manager.cc:1905] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:11.475229 7860 raft_consensus.cc:2229] T d9e021f87fd44055bf7cd647e22d13a8 P babeb35fec414bac989abcc216118c74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:11.475378 7860 raft_consensus.cc:2258] T d9e021f87fd44055bf7cd647e22d13a8 P babeb35fec414bac989abcc216118c74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:11.476277 7860 ts_tablet_manager.cc:1905] T d9e021f87fd44055bf7cd647e22d13a8 P babeb35fec414bac989abcc216118c74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:11.478101 7859 ts_tablet_manager.cc:1918] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:44:11.478257 7859 log.cc:1196] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-1/wal/wals/d9e021f87fd44055bf7cd647e22d13a8
I20260927 10:44:11.478703 7859 ts_tablet_manager.cc:1939] T d9e021f87fd44055bf7cd647e22d13a8 P ecaafcecaf3e414984d0c826a0c4246d: Deleting consensus metadata
I20260927 10:44:11.479401 7860 ts_tablet_manager.cc:1918] T d9e021f87fd44055bf7cd647e22d13a8 P babeb35fec414bac989abcc216118c74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:44:11.479485 7860 log.cc:1196] T d9e021f87fd44055bf7cd647e22d13a8 P babeb35fec414bac989abcc216118c74: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/d9e021f87fd44055bf7cd647e22d13a8
I20260927 10:44:11.479763 7860 ts_tablet_manager.cc:1939] T d9e021f87fd44055bf7cd647e22d13a8 P babeb35fec414bac989abcc216118c74: Deleting consensus metadata
I20260927 10:44:11.480403 7077 catalog_manager.cc:4841] TS ecaafcecaf3e414984d0c826a0c4246d (127.0.98.2:44599): tablet d9e021f87fd44055bf7cd647e22d13a8 (table test-table [id=71f2b3d11345453ea260c44707cbcd21]) successfully deleted
I20260927 10:44:11.480829 7078 catalog_manager.cc:4841] TS babeb35fec414bac989abcc216118c74 (127.0.98.3:44633): tablet d9e021f87fd44055bf7cd647e22d13a8 (table test-table [id=71f2b3d11345453ea260c44707cbcd21]) successfully deleted
I20260927 10:44:11.483706 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 7462
I20260927 10:44:11.484700 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:11.484392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38249 (local address 127.0.0.1:52036)
0927 10:44:11.484528 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:11.484594 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 10:44:11.484956 7075 catalog_manager.cc:4584] TS e5c1e1840c784fdd871dc6b3572c1e83 (127.0.98.1:38249): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d9e021f87fd44055bf7cd647e22d13a8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38249: connect: Connection refused (error 111)
I20260927 10:44:11.490917 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 7598
I20260927 10:44:11.497215 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 6955
I20260927 10:44:11.503293 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 7055
2026-09-27T10:44:11Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7635 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:44:11 dist-test-slave-dzhw krb5kdc[7864](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:44:11 dist-test-slave-dzhw krb5kdc[7864](info): set up 2 sockets
Sep 27 10:44:11 dist-test-slave-dzhw krb5kdc[7864](info): commencing operation
krb5kdc: starting...
W20260927 10:44:12.547024 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:44:12 dist-test-slave-dzhw krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:44:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:44:12Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 10:44:12.673698 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39745
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:12.784497 7877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:12.784842 7877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:12.784917 7877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:12.789139 7877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:44:12.789245 7877 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 10:44:12.789278 7877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:12.789330 7877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:12.789374 7877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:44:12.789409 7877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:44:12.795226 7877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39745
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:12.796413 7877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:12.797451 7877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:12.806847 7883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:12.807011 7877 server_base.cc:1034] running on GCE node
W20260927 10:44:12.806847 7882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:12.806931 7885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:12.807782 7877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:12.808967 7877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:12.810163 7877 hybrid_clock.cc:648] HybridClock initialized: now 1790505852810134 us; error 40 us; skew 500 ppm
Sep 27 10:44:12 dist-test-slave-dzhw krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505852, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:12.813218 7877 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 10:44:12.814225 7877 webserver.cc:460] Webserver started at http://127.0.98.62:36865/ using document root <none> and password file <none>
I20260927 10:44:12.814875 7877 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:12.814970 7877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:12.815241 7877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:12.817380 7877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "2ae783e79d7b4186b372344edaec95d3"
format_stamp: "Formatted at 2026-09-27 10:44:12 on dist-test-slave-dzhw"
server_key: "9d5bd477abd41ed3d531a35837c9f94c"
server_key_iv: "7b8597adbcd7229dcd0f654964480b3f"
server_key_version: "encryptionkey@0"
I20260927 10:44:12.817970 7877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "2ae783e79d7b4186b372344edaec95d3"
format_stamp: "Formatted at 2026-09-27 10:44:12 on dist-test-slave-dzhw"
server_key: "9d5bd477abd41ed3d531a35837c9f94c"
server_key_iv: "7b8597adbcd7229dcd0f654964480b3f"
server_key_version: "encryptionkey@0"
I20260927 10:44:12.821861 7877 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 10:44:12.824479 7892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:12.825788 7877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:44:12.825944 7877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "2ae783e79d7b4186b372344edaec95d3"
format_stamp: "Formatted at 2026-09-27 10:44:12 on dist-test-slave-dzhw"
server_key: "9d5bd477abd41ed3d531a35837c9f94c"
server_key_iv: "7b8597adbcd7229dcd0f654964480b3f"
server_key_version: "encryptionkey@0"
I20260927 10:44:12.826071 7877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:12.847205 7877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:12.850703 7877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:12.850912 7877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:12.858908 7877 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39745
I20260927 10:44:12.858927 7954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39745 every 8 connection(s)
I20260927 10:44:12.859890 7877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:44:12.860260 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 7877
I20260927 10:44:12.860421 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:44:12.860713 392 external_mini_cluster.cc:1428] Setting key b771fe5d81fe34f9ff1b89721de3d366
I20260927 10:44:12.863029 7955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:12.868389 7955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3: Bootstrap starting.
Sep 27 10:44:12 dist-test-slave-dzhw krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 10:44:12.870805 7955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:12.871908 7955 log.cc:824] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:12.874085 7955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3: No bootstrap required, opened a new log
I20260927 10:44:12.874733 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:12.862089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43818 (local address 127.0.98.62:39745)
0927 10:44:12.862583 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:12.862599 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:12.862637 (+ 38us) server_negotiation.cc:407] Connection header received
0927 10:44:12.863332 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:12.863349 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:12.863688 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:12.864029 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:12.864819 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:12.865803 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:12.866706 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:12.867186 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:12.867354 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:12.869864 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:12.869869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:12.869883 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:12.869910 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:12.871727 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:12.872241 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:12.872246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:12.872251 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:12.872418 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:12.872716 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:12.872721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:12.872723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:12.873108 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:12.873258 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:12.873563 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 10:44:12.873867 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":146,"threads_started":1}
I20260927 10:44:12.877547 7955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae783e79d7b4186b372344edaec95d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39745 } }
I20260927 10:44:12.877804 7955 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:12.877882 7955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae783e79d7b4186b372344edaec95d3, State: Initialized, Role: FOLLOWER
I20260927 10:44:12.878298 7955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae783e79d7b4186b372344edaec95d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39745 } }
I20260927 10:44:12.878470 7955 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:44:12.878558 7955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:44:12.878682 7955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:12.879613 7955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae783e79d7b4186b372344edaec95d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39745 } }
I20260927 10:44:12.879945 7955 leader_election.cc:302] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ae783e79d7b4186b372344edaec95d3; no voters:
I20260927 10:44:12.880255 7955 leader_election.cc:288] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:44:12.880546 7960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:12.880779 7960 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3 [term 1 LEADER]: Becoming Leader. State: Replica: 2ae783e79d7b4186b372344edaec95d3, State: Running, Role: LEADER
I20260927 10:44:12.881094 7960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae783e79d7b4186b372344edaec95d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39745 } }
I20260927 10:44:12.881639 7955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:44:12.882803 7961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ae783e79d7b4186b372344edaec95d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae783e79d7b4186b372344edaec95d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39745 } } }
I20260927 10:44:12.882933 7961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3 [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:12.883335 7962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ae783e79d7b4186b372344edaec95d3. Latest consensus state: current_term: 1 leader_uuid: "2ae783e79d7b4186b372344edaec95d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae783e79d7b4186b372344edaec95d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39745 } } }
I20260927 10:44:12.883448 7962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3 [sys.catalog]: This master's current role is: LEADER
W20260927 10:44:12.886487 7975 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:44:12.886588 7975 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:44:12.886678 7969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:44:12.887423 7969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:44:12.893656 7969 catalog_manager.cc:1347] Generated new cluster ID: 56b3dde68f8c4ac0900c55cf31c3bfbf
I20260927 10:44:12.893747 7969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:44:12.910646 7969 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:44:12.911602 7969 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:44:12.915479 7969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3: Generated new TSK 0
I20260927 10:44:12.916167 7969 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 10:44:12.950722 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39745
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:13.064229 7981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:13.064610 7981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:13.064716 7981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:13.068984 7981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:13.069056 7981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:13.069097 7981 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:44:13.069139 7981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:44:13.074409 7981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:13.075548 7981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:13.076567 7981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:13.086249 7986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:13.086403 7981 server_base.cc:1034] running on GCE node
W20260927 10:44:13.086274 7987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:13.086246 7989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:13.086907 7981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:13.087620 7981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:13.088819 7981 hybrid_clock.cc:648] HybridClock initialized: now 1790505853088803 us; error 31 us; skew 500 ppm
Sep 27 10:44:13 dist-test-slave-dzhw krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505853, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:13.091651 7981 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 10:44:13.092555 7981 webserver.cc:460] Webserver started at http://127.0.98.1:32841/ using document root <none> and password file <none>
I20260927 10:44:13.093124 7981 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:13.093216 7981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:13.093447 7981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:13.095587 7981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "c7de0b62beb8473a81706766c3aca2ab"
format_stamp: "Formatted at 2026-09-27 10:44:13 on dist-test-slave-dzhw"
server_key: "6e7f1e87343c2485d4e5b8cf7d7af7fa"
server_key_iv: "b4d84f575c62760ea8c48123c0820c32"
server_key_version: "encryptionkey@0"
I20260927 10:44:13.096172 7981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c7de0b62beb8473a81706766c3aca2ab"
format_stamp: "Formatted at 2026-09-27 10:44:13 on dist-test-slave-dzhw"
server_key: "6e7f1e87343c2485d4e5b8cf7d7af7fa"
server_key_iv: "b4d84f575c62760ea8c48123c0820c32"
server_key_version: "encryptionkey@0"
I20260927 10:44:13.099954 7981 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:44:13.102584 7996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:13.103876 7981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:44:13.104010 7981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "c7de0b62beb8473a81706766c3aca2ab"
format_stamp: "Formatted at 2026-09-27 10:44:13 on dist-test-slave-dzhw"
server_key: "6e7f1e87343c2485d4e5b8cf7d7af7fa"
server_key_iv: "b4d84f575c62760ea8c48123c0820c32"
server_key_version: "encryptionkey@0"
I20260927 10:44:13.104131 7981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:13.121065 7981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:13.124220 7981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:13.124426 7981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:13.125869 7981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:13.125943 7981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:13.126013 7981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:13.126057 7981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:44:13 dist-test-slave-dzhw krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505853, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 10:44:13.137202 7981 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37455
I20260927 10:44:13.137269 8111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37455 every 8 connection(s)
I20260927 10:44:13.138238 7981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:44:13.139438 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:13.126894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56457 (local address 127.0.98.62:39745)
0927 10:44:13.127016 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:13.127020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:13.128003 (+ 983us) server_negotiation.cc:407] Connection header received
0927 10:44:13.129041 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:13.129047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:13.129101 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:13.129196 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:13.130797 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.131608 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.132749 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.133058 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.133174 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:13.135691 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:13.135696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:13.135700 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:13.135733 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:13.137441 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:13.137905 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:13.137909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:13.137912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:13.138005 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:13.138320 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:13.138323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:13.138325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:13.138569 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:13.138687 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:13.139208 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 10:44:13.139320 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 10:44:13.140228 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:13.127408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39745 (local address 127.0.98.1:56457)
0927 10:44:13.127861 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:13.127901 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:13.128822 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:13.129402 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:13.129413 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:13.129932 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:13.130591 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:13.130615 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.131743 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.131746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:13.132612 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:13.132621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.133232 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.133237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:13.133391 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:13.133994 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:13.134011 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:13.135532 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:13.137545 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:13.137549 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:13.137560 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:13.137804 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:13.138095 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:13.138097 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:13.138098 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:13.138225 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:13.138680 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:13.138686 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:13.138949 (+ 263us) client_negotiation.cc:769] Sending connection context
0927 10:44:13.139146 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 10:44:13.139417 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":121,"threads_started":1}
I20260927 10:44:13.141330 8112 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39745
I20260927 10:44:13.141541 8112 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:13.142086 8112 heartbeater.cc:511] Master 127.0.98.62:39745 requested a full tablet report, sending...
I20260927 10:44:13.143873 7908 ts_manager.cc:194] Registered new tserver with Master: c7de0b62beb8473a81706766c3aca2ab (127.0.98.1:37455)
I20260927 10:44:13.144976 7908 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56457
I20260927 10:44:13.147384 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 7981
I20260927 10:44:13.147518 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:44:13.147832 392 external_mini_cluster.cc:1428] Setting key 445534ad1e160eaffecf92e55750ddd0
I20260927 10:44:13.152839 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:13.143863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46767 (local address 127.0.98.62:39745)
0927 10:44:13.143981 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:13.143986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:13.145745 (+ 1759us) server_negotiation.cc:407] Connection header received
0927 10:44:13.145855 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:13.145857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:13.145903 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:13.145987 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:13.146396 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.147154 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.148287 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.148537 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.148657 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:13.149814 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:13.149818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:13.149821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:13.149850 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:13.151266 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:13.151677 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:13.151683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:13.151687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:13.151764 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:13.152036 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:13.152040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:13.152042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:13.152208 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:13.152324 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:13.152568 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 10:44:13.152671 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":41}
I20260927 10:44:13.152848 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:13.143790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39745 (local address 127.0.98.1:46767)
0927 10:44:13.145650 (+ 1860us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:13.145665 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:13.145767 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:13.146008 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:13.146010 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:13.146187 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:13.146289 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:13.146293 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.147310 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.147314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:13.148158 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:13.148169 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.148658 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.148662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:13.148771 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:13.149242 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:13.149277 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:13.149685 (+ 408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:13.151377 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:13.151381 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:13.151383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:13.151579 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:13.151856 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:13.151860 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:13.151862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:13.151946 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:13.152314 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:13.152317 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:13.152406 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 10:44:13.152519 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 10:44:13.152662 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1758,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 10:44:13.177477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39745
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:13.286950 8121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:13.287240 8121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:13.287285 8121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:13.291324 8121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:13.291390 8121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:13.291429 8121 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:44:13.291492 8121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:44:13.296758 8121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:13.297840 8121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:13.299036 8121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:13.308557 8126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:13.308614 8129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:13.308557 8127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:13.308929 8121 server_base.cc:1034] running on GCE node
I20260927 10:44:13.309347 8121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:13.310006 8121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:13.311161 8121 hybrid_clock.cc:648] HybridClock initialized: now 1790505853311144 us; error 31 us; skew 500 ppm
Sep 27 10:44:13 dist-test-slave-dzhw krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505853, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:13.314054 8121 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 10:44:13.315088 8121 webserver.cc:460] Webserver started at http://127.0.98.2:38305/ using document root <none> and password file <none>
I20260927 10:44:13.315686 8121 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:13.315781 8121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:13.316018 8121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:13.318279 8121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "bef3dd5461af4e37b5bace4d2f36536a"
format_stamp: "Formatted at 2026-09-27 10:44:13 on dist-test-slave-dzhw"
server_key: "d907fb72a966e78a5a8b48faf3c0415d"
server_key_iv: "c11183ad79c3bc2b91596372dd98a5a9"
server_key_version: "encryptionkey@0"
I20260927 10:44:13.318928 8121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bef3dd5461af4e37b5bace4d2f36536a"
format_stamp: "Formatted at 2026-09-27 10:44:13 on dist-test-slave-dzhw"
server_key: "d907fb72a966e78a5a8b48faf3c0415d"
server_key_iv: "c11183ad79c3bc2b91596372dd98a5a9"
server_key_version: "encryptionkey@0"
I20260927 10:44:13.323086 8121 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:44:13.325781 8136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:13.327648 8121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:44:13.327817 8121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "bef3dd5461af4e37b5bace4d2f36536a"
format_stamp: "Formatted at 2026-09-27 10:44:13 on dist-test-slave-dzhw"
server_key: "d907fb72a966e78a5a8b48faf3c0415d"
server_key_iv: "c11183ad79c3bc2b91596372dd98a5a9"
server_key_version: "encryptionkey@0"
I20260927 10:44:13.327951 8121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:13.344928 8121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:13.348326 8121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:13.348549 8121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:13.350044 8121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:13.350124 8121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:13.350196 8121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:13.350250 8121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:44:13 dist-test-slave-dzhw krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505853, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 10:44:13.360857 8121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34385
I20260927 10:44:13.360931 8251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34385 every 8 connection(s)
I20260927 10:44:13.361932 8121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:44:13.363813 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 8121
I20260927 10:44:13.363958 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:44:13.364277 392 external_mini_cluster.cc:1428] Setting key f32dd158834ccda070a162d0d9ea6b77
I20260927 10:44:13.365135 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:13.350966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46121 (local address 127.0.98.62:39745)
0927 10:44:13.351118 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:13.351122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:13.351999 (+ 877us) server_negotiation.cc:407] Connection header received
0927 10:44:13.353108 (+ 1109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:13.353114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:13.353190 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:13.353290 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:13.355199 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.356091 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.357872 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.358245 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.358372 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:13.361407 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:13.361411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:13.361414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:13.361445 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:13.363063 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:13.363512 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:13.363514 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:13.363516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:13.363584 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:13.363996 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:13.364000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:13.364001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:13.364266 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:13.364374 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:13.364864 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 10:44:13.364968 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:44:13.365960 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:13.351369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39745 (local address 127.0.98.2:46121)
0927 10:44:13.351817 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:13.351869 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:13.352837 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:13.353761 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:13.353773 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:13.354238 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:13.355001 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:13.355020 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.356240 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.356245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:13.357747 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:13.357757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.358403 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.358405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:13.358670 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:13.359405 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:13.359427 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:13.361226 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:13.363177 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:13.363181 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:13.363192 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:13.363429 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:13.363675 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:13.363678 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:13.363679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:13.363864 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:13.364387 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:13.364393 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:13.364626 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 10:44:13.364784 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 10:44:13.365113 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":131,"threads_started":1}
I20260927 10:44:13.367144 8252 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39745
I20260927 10:44:13.367312 8252 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:13.367792 8252 heartbeater.cc:511] Master 127.0.98.62:39745 requested a full tablet report, sending...
I20260927 10:44:13.368829 7908 ts_manager.cc:194] Registered new tserver with Master: bef3dd5461af4e37b5bace4d2f36536a (127.0.98.2:34385)
I20260927 10:44:13.369441 7908 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46121
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260927 10:44:13.376964 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:13.369652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57087 (local address 127.0.98.62:39745)
0927 10:44:13.369767 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:13.369772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:13.370216 (+ 444us) server_negotiation.cc:407] Connection header received
0927 10:44:13.370388 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:13.370391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:13.370459 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:13.370527 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:13.370980 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.371487 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.372287 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.372515 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.372619 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:13.373748 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:13.373752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:13.373754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:13.373777 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:13.375300 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:13.375734 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:13.375738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:13.375739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:13.375823 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:13.376114 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:13.376120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:13.376122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:13.376338 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:13.376443 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:13.376698 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 10:44:13.376802 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 10:44:13.376978 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:13.369859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39745 (local address 127.0.98.2:57087)
0927 10:44:13.370103 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:13.370146 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:13.370280 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:13.370541 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:13.370543 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:13.370752 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:13.370868 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:13.370874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.371606 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.371609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:13.372170 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:13.372177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.372621 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.372625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:13.372718 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:13.373262 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:13.373274 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:13.373594 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:13.375414 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:13.375417 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:13.375419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:13.375620 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:13.375914 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:13.375917 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:13.375919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:13.376006 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:13.376433 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:13.376436 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:13.376533 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 10:44:13.376637 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 10:44:13.376794 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":3,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 10:44:13.393821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39745
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:13.502013 8261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:13.502305 8261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:13.502380 8261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:13.506723 8261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:13.506804 8261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:13.506868 8261 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:44:13.506950 8261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:44:13.512362 8261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:13.513437 8261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:13.514506 8261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:13.524266 8267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:13.524274 8269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:13.524333 8261 server_base.cc:1034] running on GCE node
W20260927 10:44:13.524289 8266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:13.524920 8261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:13.525641 8261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:13.526870 8261 hybrid_clock.cc:648] HybridClock initialized: now 1790505853526835 us; error 46 us; skew 500 ppm
Sep 27 10:44:13 dist-test-slave-dzhw krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505853, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:13.530055 8261 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 10:44:13.531065 8261 webserver.cc:460] Webserver started at http://127.0.98.3:36831/ using document root <none> and password file <none>
I20260927 10:44:13.531666 8261 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:13.531730 8261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:13.531898 8261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:13.533969 8261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "9c6b4e51bf3143a4a2bd020f699a3139"
format_stamp: "Formatted at 2026-09-27 10:44:13 on dist-test-slave-dzhw"
server_key: "ec640bd9360c1be2633dbae02285525d"
server_key_iv: "2af5653f5194d07bfb812428266ad2bf"
server_key_version: "encryptionkey@0"
I20260927 10:44:13.534569 8261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9c6b4e51bf3143a4a2bd020f699a3139"
format_stamp: "Formatted at 2026-09-27 10:44:13 on dist-test-slave-dzhw"
server_key: "ec640bd9360c1be2633dbae02285525d"
server_key_iv: "2af5653f5194d07bfb812428266ad2bf"
server_key_version: "encryptionkey@0"
I20260927 10:44:13.538275 8261 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:44:13.540766 8276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:13.542266 8261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:44:13.542397 8261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "9c6b4e51bf3143a4a2bd020f699a3139"
format_stamp: "Formatted at 2026-09-27 10:44:13 on dist-test-slave-dzhw"
server_key: "ec640bd9360c1be2633dbae02285525d"
server_key_iv: "2af5653f5194d07bfb812428266ad2bf"
server_key_version: "encryptionkey@0"
I20260927 10:44:13.542542 8261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:13.567554 8261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:13.570775 8261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:13.570994 8261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:13.572484 8261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:13.572533 8261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:13.572572 8261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:13.572616 8261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:44:13 dist-test-slave-dzhw krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505853, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 10:44:13.583603 8261 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36351
I20260927 10:44:13.583658 8391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36351 every 8 connection(s)
I20260927 10:44:13.584563 8261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:44:13.587795 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:13.573282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34945 (local address 127.0.98.62:39745)
0927 10:44:13.573492 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:13.573499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:13.574959 (+ 1460us) server_negotiation.cc:407] Connection header received
0927 10:44:13.576154 (+ 1195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:13.576158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:13.576218 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:13.576273 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:13.577923 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.578588 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.580466 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.580747 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.580833 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:13.584140 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:13.584144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:13.584146 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:13.584174 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:13.585702 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:13.586270 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:13.586273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:13.586275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:13.586353 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:13.586718 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:13.586721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:13.586723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:13.586905 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:13.587010 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:13.587498 (+ 488us) server_negotiation.cc:299] Negotiation successful
0927 10:44:13.587612 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 10:44:13.588532 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:13.573745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39745 (local address 127.0.98.3:34945)
0927 10:44:13.574805 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:13.574860 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:13.575904 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:13.576698 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:13.576708 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:13.577235 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:13.577779 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:13.577792 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.578721 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.578725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:13.580325 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:13.580339 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.580917 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.580923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:13.581141 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:13.581918 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:13.581946 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:13.583969 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:13.585823 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:13.585828 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:13.585842 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:13.586150 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:13.586475 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:13.586478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:13.586480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:13.586618 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:13.587007 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:13.587013 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:13.587267 (+ 254us) client_negotiation.cc:769] Sending connection context
0927 10:44:13.587444 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 10:44:13.587747 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"thread_start_us":119,"threads_started":1}
I20260927 10:44:13.589694 8392 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39745
I20260927 10:44:13.589847 8392 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:13.589973 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 8261
I20260927 10:44:13.590104 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:44:13.590405 392 external_mini_cluster.cc:1428] Setting key c64e21f31c2631c8491790ca08af7877
I20260927 10:44:13.590448 8392 heartbeater.cc:511] Master 127.0.98.62:39745 requested a full tablet report, sending...
I20260927 10:44:13.591547 7908 ts_manager.cc:194] Registered new tserver with Master: 9c6b4e51bf3143a4a2bd020f699a3139 (127.0.98.3:36351)
I20260927 10:44:13.592316 7908 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34945
I20260927 10:44:13.592406 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:44:13.600101 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:13.592324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54923 (local address 127.0.98.62:39745)
0927 10:44:13.592445 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:13.592449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:13.592558 (+ 109us) server_negotiation.cc:407] Connection header received
0927 10:44:13.592735 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:13.592738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:13.592781 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:13.592860 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:13.593451 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.594180 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.595203 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.595489 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.595603 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:13.596690 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:13.596696 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:13.596701 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:13.596727 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:13.598206 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:13.598748 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:13.598751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:13.598752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:13.598821 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:13.599135 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:13.599140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:13.599142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:13.599415 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:13.599517 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:13.599876 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 10:44:13.599987 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:44:13.600220 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:13.591995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39745 (local address 127.0.98.3:54923)
0927 10:44:13.592483 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:13.592498 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:13.592636 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:13.592896 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:13.592900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:13.593144 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:13.593320 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:13.593331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.594329 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.594332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:13.595077 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:13.595086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.595607 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.595610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:13.595681 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:13.596223 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:13.596234 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:13.596549 (+ 315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:13.598320 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:13.598326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:13.598332 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:13.598652 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:13.598912 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:13.598917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:13.598922 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:13.599031 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:13.599569 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:13.599574 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:13.599680 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 10:44:13.599799 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 10:44:13.599956 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":1,"thread_start_us":125,"threads_started":1}
Sep 27 10:44:13 dist-test-slave-dzhw krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505853, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 10:44:13 dist-test-slave-dzhw krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:13 dist-test-slave-dzhw krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 10:44:13.618152 7958 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:13.611479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43824 (local address 127.0.98.62:39745)
0927 10:44:13.611664 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:13.611668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:13.611683 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:44:13.611726 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:13.611729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:13.611773 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:13.611898 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:13.612630 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.613230 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.614095 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.614322 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.614471 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:13.618023 (+ 3552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43824: BlockingRecv error: recv got EOF from 127.0.0.1:43824 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 27 10:44:13 dist-test-slave-dzhw krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505853, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 10:44:13.628417 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:13.619796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43832 (local address 127.0.98.62:39745)
0927 10:44:13.619920 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:13.619923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:13.620067 (+ 144us) server_negotiation.cc:407] Connection header received
0927 10:44:13.620172 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:13.620174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:13.620217 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:13.620295 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:13.620910 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.621437 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.622231 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.622548 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.622630 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:13.625031 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:13.625034 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:13.625037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:13.625067 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:13.626563 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:13.626987 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:13.626990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:13.626991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:13.627084 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:13.627450 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:13.627454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:13.627455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:13.627646 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:13.627762 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:13.628047 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 10:44:13.628239 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:44:13.632313 7908 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:44:13.635426 7908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:44:13.650544 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:13.644837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37592 (local address 127.0.98.3:36351)
0927 10:44:13.646119 (+ 1282us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:13.646131 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:13.646151 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:44:13.646230 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:13.646235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:13.646470 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:13.646624 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:13.646822 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.647908 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.649136 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.649801 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.649908 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:13.650065 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:13.650241 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 10:44:13.650339 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1088,"thread_start_us":113,"threads_started":1}
I20260927 10:44:13.650532 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:13.644696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36351 (local address 127.0.0.1:37592)
0927 10:44:13.644957 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:13.644969 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:13.645055 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:13.646602 (+ 1547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:13.646605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:13.646622 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:13.646695 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:13.646701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.648001 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.648007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:13.649003 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:13.649012 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.649927 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.649931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:13.650033 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:13.650075 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 10:44:13.650200 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 10:44:13.650339 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":24,"thread_start_us":83,"threads_started":1}
I20260927 10:44:13.650862 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:13.643972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34385 (local address 127.0.0.1:40530)
0927 10:44:13.644429 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:13.644475 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:13.644600 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:13.645776 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:13.645781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:13.645803 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:13.645914 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:13.645939 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.647825 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.647831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:13.649413 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:13.649424 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.650464 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.650468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:13.650555 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:13.650602 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 10:44:13.650662 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 10:44:13.650747 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":106,"thread_start_us":105,"threads_started":1}
I20260927 10:44:13.651198 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:13.644935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50200 (local address 127.0.98.1:37455)
0927 10:44:13.645271 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:13.645278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:13.645300 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:44:13.645377 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:13.645383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:13.645596 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:13.645747 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:13.646342 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.647503 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.649566 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.650248 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.650339 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:13.650487 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:13.650955 (+ 468us) server_negotiation.cc:299] Negotiation successful
0927 10:44:13.651061 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":86,"threads_started":1}
I20260927 10:44:13.651296 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:13.644042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37455 (local address 127.0.0.1:50200)
0927 10:44:13.644450 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:13.644482 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:13.644600 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:13.646084 (+ 1484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:13.646088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:13.646106 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:13.646225 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:13.646231 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.647649 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.647652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:13.649413 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:13.649424 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.650546 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.650552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:13.650645 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:13.650674 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 10:44:13.650748 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 10:44:13.650822 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":52,"spinlock_wait_cycles":896,"thread_start_us":99,"threads_started":1}
I20260927 10:44:13.651501 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:13.644095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40530 (local address 127.0.98.2:34385)
0927 10:44:13.645345 (+ 1250us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:13.645355 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:13.645373 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:44:13.645432 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:13.645436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:13.645585 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:13.645747 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:13.646761 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.647698 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.649539 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.650313 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.651036 (+ 723us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:13.651205 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:13.651256 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 10:44:13.651362 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1160,"thread_start_us":105,"threads_started":1}
I20260927 10:44:13.654147 8309 tablet_service.cc:1467] Processing CreateTablet for tablet c570b6fa54074145ad11735b5b90b1bd (DEFAULT_TABLE table=test-table [id=4bf8aa77167e4185b5dd91a2301f008f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:13.654165 8186 tablet_service.cc:1467] Processing CreateTablet for tablet c570b6fa54074145ad11735b5b90b1bd (DEFAULT_TABLE table=test-table [id=4bf8aa77167e4185b5dd91a2301f008f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:13.654155 8046 tablet_service.cc:1467] Processing CreateTablet for tablet c570b6fa54074145ad11735b5b90b1bd (DEFAULT_TABLE table=test-table [id=4bf8aa77167e4185b5dd91a2301f008f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:13.655177 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c570b6fa54074145ad11735b5b90b1bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:13.655194 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c570b6fa54074145ad11735b5b90b1bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:13.655194 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c570b6fa54074145ad11735b5b90b1bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:13.661834 8420 tablet_bootstrap.cc:492] T c570b6fa54074145ad11735b5b90b1bd P 9c6b4e51bf3143a4a2bd020f699a3139: Bootstrap starting.
I20260927 10:44:13.663492 8421 tablet_bootstrap.cc:492] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab: Bootstrap starting.
I20260927 10:44:13.664179 8420 tablet_bootstrap.cc:654] T c570b6fa54074145ad11735b5b90b1bd P 9c6b4e51bf3143a4a2bd020f699a3139: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:13.665053 8420 log.cc:824] T c570b6fa54074145ad11735b5b90b1bd P 9c6b4e51bf3143a4a2bd020f699a3139: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:13.665584 8421 tablet_bootstrap.cc:654] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:13.665726 8422 tablet_bootstrap.cc:492] T c570b6fa54074145ad11735b5b90b1bd P bef3dd5461af4e37b5bace4d2f36536a: Bootstrap starting.
I20260927 10:44:13.666363 8421 log.cc:824] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:13.667366 8420 tablet_bootstrap.cc:492] T c570b6fa54074145ad11735b5b90b1bd P 9c6b4e51bf3143a4a2bd020f699a3139: No bootstrap required, opened a new log
I20260927 10:44:13.667580 8420 ts_tablet_manager.cc:1397] T c570b6fa54074145ad11735b5b90b1bd P 9c6b4e51bf3143a4a2bd020f699a3139: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:44:13.668192 8422 tablet_bootstrap.cc:654] T c570b6fa54074145ad11735b5b90b1bd P bef3dd5461af4e37b5bace4d2f36536a: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:13.668365 8421 tablet_bootstrap.cc:492] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab: No bootstrap required, opened a new log
I20260927 10:44:13.668540 8421 ts_tablet_manager.cc:1397] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:44:13.669023 8422 log.cc:824] T c570b6fa54074145ad11735b5b90b1bd P bef3dd5461af4e37b5bace4d2f36536a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:13.670962 8422 tablet_bootstrap.cc:492] T c570b6fa54074145ad11735b5b90b1bd P bef3dd5461af4e37b5bace4d2f36536a: No bootstrap required, opened a new log
I20260927 10:44:13.671149 8422 ts_tablet_manager.cc:1397] T c570b6fa54074145ad11735b5b90b1bd P bef3dd5461af4e37b5bace4d2f36536a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:44:13.671119 8420 raft_consensus.cc:348] T c570b6fa54074145ad11735b5b90b1bd P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:13.671416 8420 raft_consensus.cc:374] T c570b6fa54074145ad11735b5b90b1bd P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:13.671468 8420 raft_consensus.cc:729] T c570b6fa54074145ad11735b5b90b1bd P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c6b4e51bf3143a4a2bd020f699a3139, State: Initialized, Role: FOLLOWER
I20260927 10:44:13.671916 8420 consensus_queue.cc:260] T c570b6fa54074145ad11735b5b90b1bd P 9c6b4e51bf3143a4a2bd020f699a3139 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:13.672559 8392 heartbeater.cc:503] Master 127.0.98.62:39745 was elected leader, sending a full tablet report...
I20260927 10:44:13.672631 8421 raft_consensus.cc:348] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:13.672858 8421 raft_consensus.cc:374] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:13.672905 8421 raft_consensus.cc:729] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7de0b62beb8473a81706766c3aca2ab, State: Initialized, Role: FOLLOWER
I20260927 10:44:13.672927 8420 ts_tablet_manager.cc:1428] T c570b6fa54074145ad11735b5b90b1bd P 9c6b4e51bf3143a4a2bd020f699a3139: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:44:13.673295 8421 consensus_queue.cc:260] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:13.674166 8421 ts_tablet_manager.cc:1428] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260927 10:44:13.674536 8112 heartbeater.cc:503] Master 127.0.98.62:39745 was elected leader, sending a full tablet report...
I20260927 10:44:13.675104 8422 raft_consensus.cc:348] T c570b6fa54074145ad11735b5b90b1bd P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:13.675496 8422 raft_consensus.cc:374] T c570b6fa54074145ad11735b5b90b1bd P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:13.675568 8422 raft_consensus.cc:729] T c570b6fa54074145ad11735b5b90b1bd P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bef3dd5461af4e37b5bace4d2f36536a, State: Initialized, Role: FOLLOWER
I20260927 10:44:13.676016 8422 consensus_queue.cc:260] T c570b6fa54074145ad11735b5b90b1bd P bef3dd5461af4e37b5bace4d2f36536a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:13.676822 8252 heartbeater.cc:503] Master 127.0.98.62:39745 was elected leader, sending a full tablet report...
I20260927 10:44:13.676919 8422 ts_tablet_manager.cc:1428] T c570b6fa54074145ad11735b5b90b1bd P bef3dd5461af4e37b5bace4d2f36536a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:44:13.782588 8427 raft_consensus.cc:482] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:44:13.782819 8427 raft_consensus.cc:504] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:13.784134 8427 leader_election.cc:288] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bef3dd5461af4e37b5bace4d2f36536a (127.0.98.2:34385), 9c6b4e51bf3143a4a2bd020f699a3139 (127.0.98.3:36351)
I20260927 10:44:13.788158 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:13.784470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57931 (local address 127.0.98.2:34385)
0927 10:44:13.784607 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:13.784612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:13.784885 (+ 273us) server_negotiation.cc:407] Connection header received
0927 10:44:13.785028 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:13.785031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:13.785086 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:13.785172 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:13.785443 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.785973 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.787153 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.787561 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.787687 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:13.787747 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:13.787954 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 10:44:13.788024 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 10:44:13.788157 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:13.784591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37709 (local address 127.0.98.3:36351)
0927 10:44:13.784739 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:13.784743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:13.784944 (+ 201us) server_negotiation.cc:407] Connection header received
0927 10:44:13.785095 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:13.785099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:13.785150 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:13.785237 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:13.785493 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.786009 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.787161 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.787557 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.787674 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:13.787749 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:13.787944 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 10:44:13.788012 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:44:13.788204 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:13.784360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34385 (local address 127.0.98.1:57931)
0927 10:44:13.784760 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:13.784776 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:13.784931 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:13.785191 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:13.785194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:13.785210 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:13.785335 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:13.785341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.786089 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.786094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:13.787011 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:13.787020 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.787683 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.787687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:13.787823 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:13.787850 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:44:13.787937 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 10:44:13.788022 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":57,"thread_start_us":145,"threads_started":1}
I20260927 10:44:13.788208 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:13.784487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36351 (local address 127.0.98.1:37709)
0927 10:44:13.784871 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:13.784883 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:13.784973 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:13.785249 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:13.785251 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:13.785266 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:13.785386 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:13.785391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.786121 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.786127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:13.787028 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:13.787049 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.787679 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.787682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:13.787780 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:13.787841 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 10:44:13.787930 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 10:44:13.788012 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":40,"thread_start_us":114,"threads_started":1}
I20260927 10:44:13.788924 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c570b6fa54074145ad11735b5b90b1bd" candidate_uuid: "c7de0b62beb8473a81706766c3aca2ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" is_pre_election: true
I20260927 10:44:13.789283 8333 raft_consensus.cc:2454] T c570b6fa54074145ad11735b5b90b1bd P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7de0b62beb8473a81706766c3aca2ab in term 0.
I20260927 10:44:13.789773 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c570b6fa54074145ad11735b5b90b1bd" candidate_uuid: "c7de0b62beb8473a81706766c3aca2ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bef3dd5461af4e37b5bace4d2f36536a" is_pre_election: true
I20260927 10:44:13.789821 8000 leader_election.cc:302] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c6b4e51bf3143a4a2bd020f699a3139, c7de0b62beb8473a81706766c3aca2ab; no voters:
I20260927 10:44:13.790057 8206 raft_consensus.cc:2454] T c570b6fa54074145ad11735b5b90b1bd P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7de0b62beb8473a81706766c3aca2ab in term 0.
I20260927 10:44:13.790117 8427 raft_consensus.cc:2781] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:44:13.790205 8427 raft_consensus.cc:482] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:44:13.790261 8427 raft_consensus.cc:3037] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:13.791357 8427 raft_consensus.cc:504] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:13.791662 8427 leader_election.cc:288] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab [CANDIDATE]: Term 1 election: Requested vote from peers bef3dd5461af4e37b5bace4d2f36536a (127.0.98.2:34385), 9c6b4e51bf3143a4a2bd020f699a3139 (127.0.98.3:36351)
I20260927 10:44:13.792030 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c570b6fa54074145ad11735b5b90b1bd" candidate_uuid: "c7de0b62beb8473a81706766c3aca2ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bef3dd5461af4e37b5bace4d2f36536a"
I20260927 10:44:13.792129 8206 raft_consensus.cc:3037] T c570b6fa54074145ad11735b5b90b1bd P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:13.793111 8206 raft_consensus.cc:2454] T c570b6fa54074145ad11735b5b90b1bd P bef3dd5461af4e37b5bace4d2f36536a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7de0b62beb8473a81706766c3aca2ab in term 1.
I20260927 10:44:13.793386 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c570b6fa54074145ad11735b5b90b1bd" candidate_uuid: "c7de0b62beb8473a81706766c3aca2ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6b4e51bf3143a4a2bd020f699a3139"
I20260927 10:44:13.793411 7998 leader_election.cc:302] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bef3dd5461af4e37b5bace4d2f36536a, c7de0b62beb8473a81706766c3aca2ab; no voters:
I20260927 10:44:13.793537 8333 raft_consensus.cc:3037] T c570b6fa54074145ad11735b5b90b1bd P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:13.793605 8427 raft_consensus.cc:2781] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:13.793849 8427 raft_consensus.cc:686] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab [term 1 LEADER]: Becoming Leader. State: Replica: c7de0b62beb8473a81706766c3aca2ab, State: Running, Role: LEADER
I20260927 10:44:13.794224 8427 consensus_queue.cc:237] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:13.795200 8333 raft_consensus.cc:2454] T c570b6fa54074145ad11735b5b90b1bd P 9c6b4e51bf3143a4a2bd020f699a3139 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7de0b62beb8473a81706766c3aca2ab in term 1.
I20260927 10:44:13.797858 7908 catalog_manager.cc:5495] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab reported cstate change: term changed from 0 to 1, leader changed from <none> to c7de0b62beb8473a81706766c3aca2ab (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c7de0b62beb8473a81706766c3aca2ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } health_report { overall_health: UNKNOWN } } }
W20260927 10:44:13.836308 8393 tablet.cc:2369] T c570b6fa54074145ad11735b5b90b1bd P 9c6b4e51bf3143a4a2bd020f699a3139: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:44:13.840387 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:13.836653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50202 (local address 127.0.98.1:37455)
0927 10:44:13.836808 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:13.836812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:13.836830 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:44:13.836915 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:13.836918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:13.836997 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:13.837131 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:44:13.837474 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.837980 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.838888 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:13.839111 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:13.839315 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:13.839429 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:44:13.839887 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:44:13.839981 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:13.840110 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 10:44:13.840209 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 10:44:13.846489 8333 raft_consensus.cc:1261] T c570b6fa54074145ad11735b5b90b1bd P 9c6b4e51bf3143a4a2bd020f699a3139 [term 1 FOLLOWER]: Refusing update from remote peer c7de0b62beb8473a81706766c3aca2ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:13.846500 8206 raft_consensus.cc:1261] T c570b6fa54074145ad11735b5b90b1bd P bef3dd5461af4e37b5bace4d2f36536a [term 1 FOLLOWER]: Refusing update from remote peer c7de0b62beb8473a81706766c3aca2ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:13.847268 8431 consensus_queue.cc:1035] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:13.847472 8427 consensus_queue.cc:1035] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:13.857851 8436 mvcc.cc:205] Tried to move back new op lower bound from 7333911977347948544 to 7333911977144397824. Current Snapshot: MvccSnapshot[applied={T|T < 7333911977347948544 or (T in {7333911977347948544})}]
I20260927 10:44:13.863466 7908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:13.863626 7908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:13.866129 7908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3: Sending DeleteTablet for 3 replicas of tablet c570b6fa54074145ad11735b5b90b1bd
I20260927 10:44:13.866891 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet c570b6fa54074145ad11735b5b90b1bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:13 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40530
I20260927 10:44:13.867398 8444 tablet_replica.cc:330] T c570b6fa54074145ad11735b5b90b1bd P bef3dd5461af4e37b5bace4d2f36536a: stopping tablet replica
I20260927 10:44:13.867547 8046 tablet_service.cc:1514] Processing DeleteTablet for tablet c570b6fa54074145ad11735b5b90b1bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:13 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50200
I20260927 10:44:13.867707 8444 raft_consensus.cc:2229] T c570b6fa54074145ad11735b5b90b1bd P bef3dd5461af4e37b5bace4d2f36536a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:13.868006 8444 raft_consensus.cc:2258] T c570b6fa54074145ad11735b5b90b1bd P bef3dd5461af4e37b5bace4d2f36536a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:13.868016 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet c570b6fa54074145ad11735b5b90b1bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:13 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37592
I20260927 10:44:13.868188 7908 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:44:13.868657 7908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:44:13.869625 8444 ts_tablet_manager.cc:1905] T c570b6fa54074145ad11735b5b90b1bd P bef3dd5461af4e37b5bace4d2f36536a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:13.872437 8444 ts_tablet_manager.cc:1918] T c570b6fa54074145ad11735b5b90b1bd P bef3dd5461af4e37b5bace4d2f36536a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:44:13.872628 8444 log.cc:1196] T c570b6fa54074145ad11735b5b90b1bd P bef3dd5461af4e37b5bace4d2f36536a: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-1/wal/wals/c570b6fa54074145ad11735b5b90b1bd
I20260927 10:44:13.873023 8444 ts_tablet_manager.cc:1939] T c570b6fa54074145ad11735b5b90b1bd P bef3dd5461af4e37b5bace4d2f36536a: Deleting consensus metadata
I20260927 10:44:13.874048 8446 tablet_replica.cc:330] T c570b6fa54074145ad11735b5b90b1bd P 9c6b4e51bf3143a4a2bd020f699a3139: stopping tablet replica
I20260927 10:44:13.874348 8445 tablet_replica.cc:330] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab: stopping tablet replica
I20260927 10:44:13.874456 8446 raft_consensus.cc:2229] T c570b6fa54074145ad11735b5b90b1bd P 9c6b4e51bf3143a4a2bd020f699a3139 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:13.874536 8046 tablet_service.cc:1467] Processing CreateTablet for tablet 0c4fb5dab869438ba82af5889792cebd (DEFAULT_TABLE table=test-table [id=3e418e5ab8e04d72aa097139106853c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:13.874794 8446 raft_consensus.cc:2258] T c570b6fa54074145ad11735b5b90b1bd P 9c6b4e51bf3143a4a2bd020f699a3139 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:13.874828 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c4fb5dab869438ba82af5889792cebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:13.874809 8445 raft_consensus.cc:2229] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:44:13.875602 8445 raft_consensus.cc:2258] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:13.876408 8446 ts_tablet_manager.cc:1905] T c570b6fa54074145ad11735b5b90b1bd P 9c6b4e51bf3143a4a2bd020f699a3139: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:13.876458 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 0c4fb5dab869438ba82af5889792cebd (DEFAULT_TABLE table=test-table [id=3e418e5ab8e04d72aa097139106853c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:13.876690 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c4fb5dab869438ba82af5889792cebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:13.877261 8421 tablet_bootstrap.cc:492] T 0c4fb5dab869438ba82af5889792cebd P c7de0b62beb8473a81706766c3aca2ab: Bootstrap starting.
I20260927 10:44:13.878247 8421 tablet_bootstrap.cc:654] T 0c4fb5dab869438ba82af5889792cebd P c7de0b62beb8473a81706766c3aca2ab: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:13.878257 7894 catalog_manager.cc:4841] TS bef3dd5461af4e37b5bace4d2f36536a (127.0.98.2:34385): tablet c570b6fa54074145ad11735b5b90b1bd (table test-table [id=4bf8aa77167e4185b5dd91a2301f008f]) successfully deleted
I20260927 10:44:13.879069 8446 ts_tablet_manager.cc:1918] T c570b6fa54074145ad11735b5b90b1bd P 9c6b4e51bf3143a4a2bd020f699a3139: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:44:13.879164 8446 log.cc:1196] T c570b6fa54074145ad11735b5b90b1bd P 9c6b4e51bf3143a4a2bd020f699a3139: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/c570b6fa54074145ad11735b5b90b1bd
I20260927 10:44:13.879253 8186 tablet_service.cc:1467] Processing CreateTablet for tablet 0c4fb5dab869438ba82af5889792cebd (DEFAULT_TABLE table=test-table [id=3e418e5ab8e04d72aa097139106853c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:13.879499 8446 ts_tablet_manager.cc:1939] T c570b6fa54074145ad11735b5b90b1bd P 9c6b4e51bf3143a4a2bd020f699a3139: Deleting consensus metadata
I20260927 10:44:13.879508 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c4fb5dab869438ba82af5889792cebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:13.880759 7896 catalog_manager.cc:4841] TS 9c6b4e51bf3143a4a2bd020f699a3139 (127.0.98.3:36351): tablet c570b6fa54074145ad11735b5b90b1bd (table test-table [id=4bf8aa77167e4185b5dd91a2301f008f]) successfully deleted
I20260927 10:44:13.881179 8420 tablet_bootstrap.cc:492] T 0c4fb5dab869438ba82af5889792cebd P 9c6b4e51bf3143a4a2bd020f699a3139: Bootstrap starting.
I20260927 10:44:13.881839 8422 tablet_bootstrap.cc:492] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a: Bootstrap starting.
I20260927 10:44:13.882153 8420 tablet_bootstrap.cc:654] T 0c4fb5dab869438ba82af5889792cebd P 9c6b4e51bf3143a4a2bd020f699a3139: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:13.882773 8422 tablet_bootstrap.cc:654] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:13.883491 8445 ts_tablet_manager.cc:1905] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:13.884682 8421 tablet_bootstrap.cc:492] T 0c4fb5dab869438ba82af5889792cebd P c7de0b62beb8473a81706766c3aca2ab: No bootstrap required, opened a new log
I20260927 10:44:13.884773 8421 ts_tablet_manager.cc:1397] T 0c4fb5dab869438ba82af5889792cebd P c7de0b62beb8473a81706766c3aca2ab: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260927 10:44:13.885214 8421 raft_consensus.cc:348] T 0c4fb5dab869438ba82af5889792cebd P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:13.885334 8421 raft_consensus.cc:374] T 0c4fb5dab869438ba82af5889792cebd P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:13.885365 8421 raft_consensus.cc:729] T 0c4fb5dab869438ba82af5889792cebd P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7de0b62beb8473a81706766c3aca2ab, State: Initialized, Role: FOLLOWER
I20260927 10:44:13.885483 8421 consensus_queue.cc:260] T 0c4fb5dab869438ba82af5889792cebd P c7de0b62beb8473a81706766c3aca2ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:13.885753 8421 ts_tablet_manager.cc:1428] T 0c4fb5dab869438ba82af5889792cebd P c7de0b62beb8473a81706766c3aca2ab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:44:13.886344 8445 ts_tablet_manager.cc:1918] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:44:13.886462 8445 log.cc:1196] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-0/wal/wals/c570b6fa54074145ad11735b5b90b1bd
I20260927 10:44:13.886642 8420 tablet_bootstrap.cc:492] T 0c4fb5dab869438ba82af5889792cebd P 9c6b4e51bf3143a4a2bd020f699a3139: No bootstrap required, opened a new log
I20260927 10:44:13.886734 8420 ts_tablet_manager.cc:1397] T 0c4fb5dab869438ba82af5889792cebd P 9c6b4e51bf3143a4a2bd020f699a3139: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 10:44:13.886831 8445 ts_tablet_manager.cc:1939] T c570b6fa54074145ad11735b5b90b1bd P c7de0b62beb8473a81706766c3aca2ab: Deleting consensus metadata
I20260927 10:44:13.887193 8420 raft_consensus.cc:348] T 0c4fb5dab869438ba82af5889792cebd P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:13.887310 8420 raft_consensus.cc:374] T 0c4fb5dab869438ba82af5889792cebd P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:13.887341 8420 raft_consensus.cc:729] T 0c4fb5dab869438ba82af5889792cebd P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c6b4e51bf3143a4a2bd020f699a3139, State: Initialized, Role: FOLLOWER
I20260927 10:44:13.887451 8420 consensus_queue.cc:260] T 0c4fb5dab869438ba82af5889792cebd P 9c6b4e51bf3143a4a2bd020f699a3139 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:13.887688 8420 ts_tablet_manager.cc:1428] T 0c4fb5dab869438ba82af5889792cebd P 9c6b4e51bf3143a4a2bd020f699a3139: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 10:44:13.888118 7896 catalog_manager.cc:4841] TS c7de0b62beb8473a81706766c3aca2ab (127.0.98.1:37455): tablet c570b6fa54074145ad11735b5b90b1bd (table test-table [id=4bf8aa77167e4185b5dd91a2301f008f]) successfully deleted
I20260927 10:44:13.888743 8422 tablet_bootstrap.cc:492] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a: No bootstrap required, opened a new log
I20260927 10:44:13.888854 8422 ts_tablet_manager.cc:1397] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260927 10:44:13.889403 8422 raft_consensus.cc:348] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:13.889520 8422 raft_consensus.cc:374] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:13.889554 8422 raft_consensus.cc:729] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bef3dd5461af4e37b5bace4d2f36536a, State: Initialized, Role: FOLLOWER
I20260927 10:44:13.889685 8422 consensus_queue.cc:260] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
W20260927 10:44:13.889746 8113 tablet.cc:2369] T 0c4fb5dab869438ba82af5889792cebd P c7de0b62beb8473a81706766c3aca2ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:44:13.889947 8422 ts_tablet_manager.cc:1428] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:44:14.088867 8428 raft_consensus.cc:482] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:44:14.089049 8428 raft_consensus.cc:504] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.090252 8428 leader_election.cc:288] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7de0b62beb8473a81706766c3aca2ab (127.0.98.1:37455), 9c6b4e51bf3143a4a2bd020f699a3139 (127.0.98.3:36351)
I20260927 10:44:14.094278 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:14.090708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58189 (local address 127.0.98.1:37455)
0927 10:44:14.090871 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:14.090876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:14.090995 (+ 119us) server_negotiation.cc:407] Connection header received
0927 10:44:14.091280 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:14.091282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:14.091323 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:14.091418 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:14.091709 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.092210 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.093281 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.093679 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.093813 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:14.093886 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:14.094081 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 10:44:14.094150 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 10:44:14.094310 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:14.090586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37455 (local address 127.0.98.2:58189)
0927 10:44:14.090884 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:14.090901 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:14.091055 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:14.091433 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:14.091436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:14.091454 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:14.091601 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:14.091606 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.092322 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.092327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:14.093144 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:14.093153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.093833 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.093837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:14.093944 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:14.093966 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:44:14.094065 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 10:44:14.094139 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"spinlock_wait_cycles":43904,"thread_start_us":88,"threads_started":1}
I20260927 10:44:14.094480 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:14.090983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36351 (local address 127.0.98.2:44251)
0927 10:44:14.091330 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:14.091351 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:14.091516 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:14.091793 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:14.091796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:14.091811 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:14.091927 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:14.091932 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.092635 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.092638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:14.093369 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:14.093377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.094036 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.094039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:14.094134 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:14.094157 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:44:14.094245 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 10:44:14.094313 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":52,"threads_started":1}
I20260927 10:44:14.094465 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:14.091078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44251 (local address 127.0.98.3:36351)
0927 10:44:14.091210 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:14.091213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:14.091449 (+ 236us) server_negotiation.cc:407] Connection header received
0927 10:44:14.091648 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:14.091650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:14.091700 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:14.091786 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:14.092028 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.092521 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.093482 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.093928 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.094034 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:14.094093 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:14.094254 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 10:44:14.094317 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:44:14.094977 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4fb5dab869438ba82af5889792cebd" candidate_uuid: "bef3dd5461af4e37b5bace4d2f36536a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" is_pre_election: true
I20260927 10:44:14.095054 8066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4fb5dab869438ba82af5889792cebd" candidate_uuid: "bef3dd5461af4e37b5bace4d2f36536a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7de0b62beb8473a81706766c3aca2ab" is_pre_election: true
I20260927 10:44:14.095144 8333 raft_consensus.cc:2454] T 0c4fb5dab869438ba82af5889792cebd P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bef3dd5461af4e37b5bace4d2f36536a in term 0.
I20260927 10:44:14.095307 8066 raft_consensus.cc:2454] T 0c4fb5dab869438ba82af5889792cebd P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bef3dd5461af4e37b5bace4d2f36536a in term 0.
I20260927 10:44:14.095551 8140 leader_election.cc:302] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c6b4e51bf3143a4a2bd020f699a3139, bef3dd5461af4e37b5bace4d2f36536a; no voters:
I20260927 10:44:14.095883 8428 raft_consensus.cc:2781] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:44:14.095949 8428 raft_consensus.cc:482] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:44:14.095978 8428 raft_consensus.cc:3037] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:14.097061 8428 raft_consensus.cc:504] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.097425 8428 leader_election.cc:288] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a [CANDIDATE]: Term 1 election: Requested vote from peers c7de0b62beb8473a81706766c3aca2ab (127.0.98.1:37455), 9c6b4e51bf3143a4a2bd020f699a3139 (127.0.98.3:36351)
I20260927 10:44:14.097849 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4fb5dab869438ba82af5889792cebd" candidate_uuid: "bef3dd5461af4e37b5bace4d2f36536a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6b4e51bf3143a4a2bd020f699a3139"
I20260927 10:44:14.097956 8333 raft_consensus.cc:3037] T 0c4fb5dab869438ba82af5889792cebd P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:14.098585 8066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4fb5dab869438ba82af5889792cebd" candidate_uuid: "bef3dd5461af4e37b5bace4d2f36536a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7de0b62beb8473a81706766c3aca2ab"
I20260927 10:44:14.098740 8066 raft_consensus.cc:3037] T 0c4fb5dab869438ba82af5889792cebd P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:14.098904 8333 raft_consensus.cc:2454] T 0c4fb5dab869438ba82af5889792cebd P 9c6b4e51bf3143a4a2bd020f699a3139 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bef3dd5461af4e37b5bace4d2f36536a in term 1.
I20260927 10:44:14.099170 8140 leader_election.cc:302] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c6b4e51bf3143a4a2bd020f699a3139, bef3dd5461af4e37b5bace4d2f36536a; no voters:
I20260927 10:44:14.099351 8428 raft_consensus.cc:2781] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:14.099550 8428 raft_consensus.cc:686] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a [term 1 LEADER]: Becoming Leader. State: Replica: bef3dd5461af4e37b5bace4d2f36536a, State: Running, Role: LEADER
I20260927 10:44:14.099813 8428 consensus_queue.cc:237] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.100009 8066 raft_consensus.cc:2454] T 0c4fb5dab869438ba82af5889792cebd P c7de0b62beb8473a81706766c3aca2ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bef3dd5461af4e37b5bace4d2f36536a in term 1.
I20260927 10:44:14.102188 7908 catalog_manager.cc:5495] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a reported cstate change: term changed from 0 to 1, leader changed from <none> to bef3dd5461af4e37b5bace4d2f36536a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bef3dd5461af4e37b5bace4d2f36536a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } health_report { overall_health: UNKNOWN } } }
W20260927 10:44:14.114141 8253 tablet.cc:2369] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 10:44:14 dist-test-slave-dzhw krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505852, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 10:44:14.153478 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:14.144120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39745 (local address 127.0.0.1:43838)
0927 10:44:14.144408 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:14.144425 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:14.144522 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:14.144807 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:14.144810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:14.145235 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:14.145347 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:14.145354 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.146462 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.146466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:14.147117 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:14.147126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.147709 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.147712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:14.147777 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:14.148434 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:14.148446 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:14.150028 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:14.152013 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:14.152021 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:14.152026 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:14.152282 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:14.152544 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:14.152546 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:14.152547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:14.152620 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:14.153001 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:14.153006 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:14.153107 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 10:44:14.153195 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 10:44:14.153342 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":106,"threads_started":1}
I20260927 10:44:14.153605 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:14.144203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43838 (local address 127.0.98.62:39745)
0927 10:44:14.144454 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:14.144463 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:14.144482 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:44:14.144621 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:14.144624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:14.144671 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:14.144770 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:14.145463 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.146301 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.147246 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.147588 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.147692 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:14.150162 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:14.150167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:14.150169 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:14.150199 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:14.151897 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:14.152374 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:14.152377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:14.152379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:14.152463 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:14.152698 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:14.152700 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:14.152702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:14.152909 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:14.152997 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:14.153268 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 10:44:14.153416 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":187,"thread_start_us":92,"threads_started":1}
I20260927 10:44:14.157557 7908 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43838:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 10:44:14.158314 7908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:44:14.163094 8046 tablet_service.cc:1467] Processing CreateTablet for tablet 883bea59114d48be92899415f157a9f4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f81d700762824b02af11ab1b9be7633f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:44:14.163232 8186 tablet_service.cc:1467] Processing CreateTablet for tablet 883bea59114d48be92899415f157a9f4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f81d700762824b02af11ab1b9be7633f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:44:14.163550 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 883bea59114d48be92899415f157a9f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:14.163475 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 883bea59114d48be92899415f157a9f4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f81d700762824b02af11ab1b9be7633f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:44:14.163604 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 883bea59114d48be92899415f157a9f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:14.163805 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 883bea59114d48be92899415f157a9f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:14.166000 8422 tablet_bootstrap.cc:492] T 883bea59114d48be92899415f157a9f4 P bef3dd5461af4e37b5bace4d2f36536a: Bootstrap starting.
I20260927 10:44:14.166632 8420 tablet_bootstrap.cc:492] T 883bea59114d48be92899415f157a9f4 P 9c6b4e51bf3143a4a2bd020f699a3139: Bootstrap starting.
I20260927 10:44:14.167122 8421 tablet_bootstrap.cc:492] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab: Bootstrap starting.
I20260927 10:44:14.167135 8422 tablet_bootstrap.cc:654] T 883bea59114d48be92899415f157a9f4 P bef3dd5461af4e37b5bace4d2f36536a: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:14.167624 8420 tablet_bootstrap.cc:654] T 883bea59114d48be92899415f157a9f4 P 9c6b4e51bf3143a4a2bd020f699a3139: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:14.167960 8421 tablet_bootstrap.cc:654] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:14.168547 8422 tablet_bootstrap.cc:492] T 883bea59114d48be92899415f157a9f4 P bef3dd5461af4e37b5bace4d2f36536a: No bootstrap required, opened a new log
I20260927 10:44:14.168654 8422 ts_tablet_manager.cc:1397] T 883bea59114d48be92899415f157a9f4 P bef3dd5461af4e37b5bace4d2f36536a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:44:14.168736 8420 tablet_bootstrap.cc:492] T 883bea59114d48be92899415f157a9f4 P 9c6b4e51bf3143a4a2bd020f699a3139: No bootstrap required, opened a new log
I20260927 10:44:14.168820 8420 ts_tablet_manager.cc:1397] T 883bea59114d48be92899415f157a9f4 P 9c6b4e51bf3143a4a2bd020f699a3139: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 10:44:14.169121 8422 raft_consensus.cc:348] T 883bea59114d48be92899415f157a9f4 P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.169214 8421 tablet_bootstrap.cc:492] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab: No bootstrap required, opened a new log
I20260927 10:44:14.169170 8420 raft_consensus.cc:348] T 883bea59114d48be92899415f157a9f4 P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.169231 8422 raft_consensus.cc:374] T 883bea59114d48be92899415f157a9f4 P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:14.169262 8420 raft_consensus.cc:374] T 883bea59114d48be92899415f157a9f4 P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:14.169284 8420 raft_consensus.cc:729] T 883bea59114d48be92899415f157a9f4 P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c6b4e51bf3143a4a2bd020f699a3139, State: Initialized, Role: FOLLOWER
I20260927 10:44:14.169276 8422 raft_consensus.cc:729] T 883bea59114d48be92899415f157a9f4 P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bef3dd5461af4e37b5bace4d2f36536a, State: Initialized, Role: FOLLOWER
I20260927 10:44:14.169291 8421 ts_tablet_manager.cc:1397] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:44:14.169420 8420 consensus_queue.cc:260] T 883bea59114d48be92899415f157a9f4 P 9c6b4e51bf3143a4a2bd020f699a3139 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.169723 8420 ts_tablet_manager.cc:1428] T 883bea59114d48be92899415f157a9f4 P 9c6b4e51bf3143a4a2bd020f699a3139: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:44:14.169663 8421 raft_consensus.cc:348] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.169658 8426 tablet_replica.cc:439] T 883bea59114d48be92899415f157a9f4 P 9c6b4e51bf3143a4a2bd020f699a3139: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } } }
I20260927 10:44:14.169777 8421 raft_consensus.cc:374] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:14.169808 8426 tablet_replica.cc:442] T 883bea59114d48be92899415f157a9f4 P 9c6b4e51bf3143a4a2bd020f699a3139: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:44:14.169811 8421 raft_consensus.cc:729] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7de0b62beb8473a81706766c3aca2ab, State: Initialized, Role: FOLLOWER
I20260927 10:44:14.169920 8421 consensus_queue.cc:260] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.170074 8422 consensus_queue.cc:260] T 883bea59114d48be92899415f157a9f4 P bef3dd5461af4e37b5bace4d2f36536a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.170198 8421 ts_tablet_manager.cc:1428] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:44:14.170359 8422 ts_tablet_manager.cc:1428] T 883bea59114d48be92899415f157a9f4 P bef3dd5461af4e37b5bace4d2f36536a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:44:14.170337 8428 tablet_replica.cc:439] T 883bea59114d48be92899415f157a9f4 P bef3dd5461af4e37b5bace4d2f36536a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } } }
I20260927 10:44:14.170506 8428 tablet_replica.cc:442] T 883bea59114d48be92899415f157a9f4 P bef3dd5461af4e37b5bace4d2f36536a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:44:14.170435 8441 tablet_replica.cc:439] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } } }
I20260927 10:44:14.170567 8441 tablet_replica.cc:442] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:44:14.231835 8441 raft_consensus.cc:482] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:44:14.232007 8441 raft_consensus.cc:504] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.232487 8441 leader_election.cc:288] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bef3dd5461af4e37b5bace4d2f36536a (127.0.98.2:34385), 9c6b4e51bf3143a4a2bd020f699a3139 (127.0.98.3:36351)
I20260927 10:44:14.232934 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883bea59114d48be92899415f157a9f4" candidate_uuid: "c7de0b62beb8473a81706766c3aca2ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bef3dd5461af4e37b5bace4d2f36536a" is_pre_election: true
I20260927 10:44:14.232985 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883bea59114d48be92899415f157a9f4" candidate_uuid: "c7de0b62beb8473a81706766c3aca2ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" is_pre_election: true
I20260927 10:44:14.233148 8333 raft_consensus.cc:2454] T 883bea59114d48be92899415f157a9f4 P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7de0b62beb8473a81706766c3aca2ab in term 0.
I20260927 10:44:14.233168 8206 raft_consensus.cc:2454] T 883bea59114d48be92899415f157a9f4 P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7de0b62beb8473a81706766c3aca2ab in term 0.
I20260927 10:44:14.233448 8000 leader_election.cc:302] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c6b4e51bf3143a4a2bd020f699a3139, c7de0b62beb8473a81706766c3aca2ab; no voters:
I20260927 10:44:14.233651 8441 raft_consensus.cc:2781] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:44:14.233738 8441 raft_consensus.cc:482] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:44:14.233791 8441 raft_consensus.cc:3037] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:14.234838 8441 raft_consensus.cc:504] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.235198 8441 leader_election.cc:288] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab [CANDIDATE]: Term 1 election: Requested vote from peers bef3dd5461af4e37b5bace4d2f36536a (127.0.98.2:34385), 9c6b4e51bf3143a4a2bd020f699a3139 (127.0.98.3:36351)
I20260927 10:44:14.235605 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883bea59114d48be92899415f157a9f4" candidate_uuid: "c7de0b62beb8473a81706766c3aca2ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6b4e51bf3143a4a2bd020f699a3139"
I20260927 10:44:14.235746 8333 raft_consensus.cc:3037] T 883bea59114d48be92899415f157a9f4 P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:14.235737 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883bea59114d48be92899415f157a9f4" candidate_uuid: "c7de0b62beb8473a81706766c3aca2ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bef3dd5461af4e37b5bace4d2f36536a"
I20260927 10:44:14.235893 8206 raft_consensus.cc:3037] T 883bea59114d48be92899415f157a9f4 P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:14.236758 8333 raft_consensus.cc:2454] T 883bea59114d48be92899415f157a9f4 P 9c6b4e51bf3143a4a2bd020f699a3139 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7de0b62beb8473a81706766c3aca2ab in term 1.
I20260927 10:44:14.236788 8206 raft_consensus.cc:2454] T 883bea59114d48be92899415f157a9f4 P bef3dd5461af4e37b5bace4d2f36536a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7de0b62beb8473a81706766c3aca2ab in term 1.
I20260927 10:44:14.237062 8000 leader_election.cc:302] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c6b4e51bf3143a4a2bd020f699a3139, c7de0b62beb8473a81706766c3aca2ab; no voters:
I20260927 10:44:14.237265 8441 raft_consensus.cc:2781] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:14.237401 8441 raft_consensus.cc:686] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab [term 1 LEADER]: Becoming Leader. State: Replica: c7de0b62beb8473a81706766c3aca2ab, State: Running, Role: LEADER
I20260927 10:44:14.237545 8441 consensus_queue.cc:237] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.238252 8427 tablet_replica.cc:439] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab: TxnStatusTablet state changed. Reason: New leader c7de0b62beb8473a81706766c3aca2ab. Latest consensus state: current_term: 1 leader_uuid: "c7de0b62beb8473a81706766c3aca2ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } } }
I20260927 10:44:14.238368 8427 tablet_replica.cc:442] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:44:14.238683 8465 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 10:44:14.238853 8465 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 10:44:14.239355 7909 catalog_manager.cc:5495] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab reported cstate change: term changed from 0 to 1, leader changed from <none> to c7de0b62beb8473a81706766c3aca2ab (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c7de0b62beb8473a81706766c3aca2ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } health_report { overall_health: UNKNOWN } } }
I20260927 10:44:14.256613 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:14.252899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37455 (local address 127.0.0.1:50218)
0927 10:44:14.253133 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:14.253153 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:14.253270 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:14.253542 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:14.253545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:14.253552 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:44:14.253648 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:14.253653 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.254692 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.254695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:14.255327 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:14.255335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.255939 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.255942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:14.256020 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:14.256069 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:44:14.256367 (+ 298us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:44:14.256377 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 10:44:14.256430 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 10:44:14.256497 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 10:44:14.256659 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:14.253132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50218 (local address 127.0.98.1:37455)
0927 10:44:14.253278 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:14.253283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:14.253300 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:44:14.253353 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:14.253356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:14.253409 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:14.253492 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:44:14.253756 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.254570 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.255438 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.255788 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.255889 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:14.256202 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:44:14.256285 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:44:14.256355 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:14.256463 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 10:44:14.256516 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:44:14.259194 8333 raft_consensus.cc:1261] T 883bea59114d48be92899415f157a9f4 P 9c6b4e51bf3143a4a2bd020f699a3139 [term 1 FOLLOWER]: Refusing update from remote peer c7de0b62beb8473a81706766c3aca2ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:14.259322 8206 raft_consensus.cc:1261] T 883bea59114d48be92899415f157a9f4 P bef3dd5461af4e37b5bace4d2f36536a [term 1 FOLLOWER]: Refusing update from remote peer c7de0b62beb8473a81706766c3aca2ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:14.259635 8441 consensus_queue.cc:1035] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:14.259774 8427 consensus_queue.cc:1035] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:14.261617 8426 tablet_replica.cc:439] T 883bea59114d48be92899415f157a9f4 P 9c6b4e51bf3143a4a2bd020f699a3139: TxnStatusTablet state changed. Reason: New leader c7de0b62beb8473a81706766c3aca2ab. Latest consensus state: current_term: 1 leader_uuid: "c7de0b62beb8473a81706766c3aca2ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } } }
I20260927 10:44:14.261736 8426 tablet_replica.cc:442] T 883bea59114d48be92899415f157a9f4 P 9c6b4e51bf3143a4a2bd020f699a3139: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:44:14.262117 8428 tablet_replica.cc:439] T 883bea59114d48be92899415f157a9f4 P bef3dd5461af4e37b5bace4d2f36536a: TxnStatusTablet state changed. Reason: New leader c7de0b62beb8473a81706766c3aca2ab. Latest consensus state: current_term: 1 leader_uuid: "c7de0b62beb8473a81706766c3aca2ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } } }
I20260927 10:44:14.262218 8428 tablet_replica.cc:442] T 883bea59114d48be92899415f157a9f4 P bef3dd5461af4e37b5bace4d2f36536a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:44:14.263401 8427 tablet_replica.cc:439] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c7de0b62beb8473a81706766c3aca2ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } } }
I20260927 10:44:14.263515 8427 tablet_replica.cc:442] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:44:14.263996 8434 mvcc.cc:205] Tried to move back new op lower bound from 7333911979039825920 to 7333911978956967936. Current Snapshot: MvccSnapshot[applied={T|T < 7333911979039825920 or (T in {7333911979039825920})}]
I20260927 10:44:14.264359 8435 mvcc.cc:205] Tried to move back new op lower bound from 7333911979039825920 to 7333911978956967936. Current Snapshot: MvccSnapshot[applied={T|T < 7333911979039825920}]
I20260927 10:44:14.266141 8428 tablet_replica.cc:439] T 883bea59114d48be92899415f157a9f4 P bef3dd5461af4e37b5bace4d2f36536a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c7de0b62beb8473a81706766c3aca2ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } } }
I20260927 10:44:14.266251 8428 tablet_replica.cc:442] T 883bea59114d48be92899415f157a9f4 P bef3dd5461af4e37b5bace4d2f36536a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:44:14.267592 8431 tablet_replica.cc:439] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c7de0b62beb8473a81706766c3aca2ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } } }
I20260927 10:44:14.267714 8431 tablet_replica.cc:442] T 883bea59114d48be92899415f157a9f4 P c7de0b62beb8473a81706766c3aca2ab: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:44:14.268607 8426 tablet_replica.cc:439] T 883bea59114d48be92899415f157a9f4 P 9c6b4e51bf3143a4a2bd020f699a3139: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c7de0b62beb8473a81706766c3aca2ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } } }
I20260927 10:44:14.268741 8426 tablet_replica.cc:442] T 883bea59114d48be92899415f157a9f4 P 9c6b4e51bf3143a4a2bd020f699a3139: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:44:14.272697 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:14.269276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40534 (local address 127.0.98.2:34385)
0927 10:44:14.269387 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:14.269391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:14.269405 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:44:14.269474 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:14.269476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:14.269523 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:14.269643 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:44:14.269877 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.270385 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.271250 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.271485 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.271593 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:14.271865 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:44:14.272313 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:44:14.272439 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:14.272503 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 10:44:14.272556 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 10:44:14.283243 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:14.279624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33405 (local address 127.0.98.1:37455)
0927 10:44:14.279739 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:14.279743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:14.279958 (+ 215us) server_negotiation.cc:407] Connection header received
0927 10:44:14.280080 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:14.280083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:14.280124 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:14.280188 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:44:14.280446 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.280922 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.282127 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.282437 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.282576 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:14.282861 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:44:14.282927 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:44:14.282978 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:14.283090 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 10:44:14.283134 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:44:14.283322 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:14.279545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37455 (local address 127.0.98.2:33405)
0927 10:44:14.279878 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:14.279891 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:14.279990 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:14.280220 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:14.280223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:14.280231 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:44:14.280350 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:14.280356 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.281019 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.281022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:14.282022 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:14.282032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.282593 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.282597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:14.282683 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:14.282734 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:44:14.283005 (+ 271us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:44:14.283011 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:44:14.283085 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 10:44:14.283160 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":88,"threads_started":1}
I20260927 10:44:14.290500 8333 raft_consensus.cc:1261] T 0c4fb5dab869438ba82af5889792cebd P 9c6b4e51bf3143a4a2bd020f699a3139 [term 1 FOLLOWER]: Refusing update from remote peer bef3dd5461af4e37b5bace4d2f36536a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:14.290771 8066 raft_consensus.cc:1261] T 0c4fb5dab869438ba82af5889792cebd P c7de0b62beb8473a81706766c3aca2ab [term 1 FOLLOWER]: Refusing update from remote peer bef3dd5461af4e37b5bace4d2f36536a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:14.290959 8428 consensus_queue.cc:1035] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:14.291280 8452 consensus_queue.cc:1035] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:14.313488 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:14.309649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35219 (local address 127.0.98.2:34385)
0927 10:44:14.309780 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:14.309785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:14.309981 (+ 196us) server_negotiation.cc:407] Connection header received
0927 10:44:14.310141 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:14.310144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:14.310200 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:14.310284 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:44:14.310661 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.311178 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.312347 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.312653 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.312769 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:14.313043 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:44:14.313122 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:44:14.313207 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:14.313297 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 10:44:14.313344 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:44:14.313501 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:14.309584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34385 (local address 127.0.98.1:35219)
0927 10:44:14.309901 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:14.309913 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:14.310032 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:14.310402 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:14.310405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:14.310412 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:44:14.310551 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:14.310557 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.311291 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.311295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:14.312204 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:14.312213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.312769 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.312772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:14.312856 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:14.312911 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:44:14.313209 (+ 298us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:44:14.313217 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 10:44:14.313279 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 10:44:14.313355 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":118,"threads_started":1}
I20260927 10:44:14.337285 7907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:14.337483 7907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:14.339711 7907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3: Sending DeleteTablet for 3 replicas of tablet 0c4fb5dab869438ba82af5889792cebd
I20260927 10:44:14.340500 8046 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c4fb5dab869438ba82af5889792cebd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50200
I20260927 10:44:14.340612 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c4fb5dab869438ba82af5889792cebd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37592
I20260927 10:44:14.340705 8445 tablet_replica.cc:330] T 0c4fb5dab869438ba82af5889792cebd P c7de0b62beb8473a81706766c3aca2ab: stopping tablet replica
I20260927 10:44:14.340807 8446 tablet_replica.cc:330] T 0c4fb5dab869438ba82af5889792cebd P 9c6b4e51bf3143a4a2bd020f699a3139: stopping tablet replica
I20260927 10:44:14.340826 8445 raft_consensus.cc:2229] T 0c4fb5dab869438ba82af5889792cebd P c7de0b62beb8473a81706766c3aca2ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:14.340912 8446 raft_consensus.cc:2229] T 0c4fb5dab869438ba82af5889792cebd P 9c6b4e51bf3143a4a2bd020f699a3139 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:14.340943 8445 raft_consensus.cc:2258] T 0c4fb5dab869438ba82af5889792cebd P c7de0b62beb8473a81706766c3aca2ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:14.341018 8446 raft_consensus.cc:2258] T 0c4fb5dab869438ba82af5889792cebd P 9c6b4e51bf3143a4a2bd020f699a3139 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:14.341583 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c4fb5dab869438ba82af5889792cebd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40530
I20260927 10:44:14.341892 8444 tablet_replica.cc:330] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a: stopping tablet replica
I20260927 10:44:14.342000 8444 raft_consensus.cc:2229] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:44:14.342314 8444 raft_consensus.cc:2258] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:14.343276 8445 ts_tablet_manager.cc:1905] T 0c4fb5dab869438ba82af5889792cebd P c7de0b62beb8473a81706766c3aca2ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:14.343461 8446 ts_tablet_manager.cc:1905] T 0c4fb5dab869438ba82af5889792cebd P 9c6b4e51bf3143a4a2bd020f699a3139: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:14.346166 8445 ts_tablet_manager.cc:1918] T 0c4fb5dab869438ba82af5889792cebd P c7de0b62beb8473a81706766c3aca2ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:44:14.346256 8445 log.cc:1196] T 0c4fb5dab869438ba82af5889792cebd P c7de0b62beb8473a81706766c3aca2ab: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-0/wal/wals/0c4fb5dab869438ba82af5889792cebd
I20260927 10:44:14.346534 8445 ts_tablet_manager.cc:1939] T 0c4fb5dab869438ba82af5889792cebd P c7de0b62beb8473a81706766c3aca2ab: Deleting consensus metadata
I20260927 10:44:14.346979 8444 ts_tablet_manager.cc:1905] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:14.347410 7896 catalog_manager.cc:4841] TS c7de0b62beb8473a81706766c3aca2ab (127.0.98.1:37455): tablet 0c4fb5dab869438ba82af5889792cebd (table test-table [id=3e418e5ab8e04d72aa097139106853c5]) successfully deleted
I20260927 10:44:14.349737 8444 ts_tablet_manager.cc:1918] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:44:14.349809 8444 log.cc:1196] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-1/wal/wals/0c4fb5dab869438ba82af5889792cebd
I20260927 10:44:14.350054 8444 ts_tablet_manager.cc:1939] T 0c4fb5dab869438ba82af5889792cebd P bef3dd5461af4e37b5bace4d2f36536a: Deleting consensus metadata
I20260927 10:44:14.350988 7894 catalog_manager.cc:4841] TS bef3dd5461af4e37b5bace4d2f36536a (127.0.98.2:34385): tablet 0c4fb5dab869438ba82af5889792cebd (table test-table [id=3e418e5ab8e04d72aa097139106853c5]) successfully deleted
I20260927 10:44:14.354249 8446 ts_tablet_manager.cc:1918] T 0c4fb5dab869438ba82af5889792cebd P 9c6b4e51bf3143a4a2bd020f699a3139: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:44:14.354346 8446 log.cc:1196] T 0c4fb5dab869438ba82af5889792cebd P 9c6b4e51bf3143a4a2bd020f699a3139: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/0c4fb5dab869438ba82af5889792cebd
I20260927 10:44:14.354636 8446 ts_tablet_manager.cc:1939] T 0c4fb5dab869438ba82af5889792cebd P 9c6b4e51bf3143a4a2bd020f699a3139: Deleting consensus metadata
I20260927 10:44:14.355387 7896 catalog_manager.cc:4841] TS 9c6b4e51bf3143a4a2bd020f699a3139 (127.0.98.3:36351): tablet 0c4fb5dab869438ba82af5889792cebd (table test-table [id=3e418e5ab8e04d72aa097139106853c5]) successfully deleted
Sep 27 10:44:14 dist-test-slave-dzhw krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 10:44:14 dist-test-slave-dzhw krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:14 dist-test-slave-dzhw krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 10:44:14.388191 8460 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:14.379120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55306 (local address 127.0.98.62:39745)
0927 10:44:14.379279 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:14.379283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:14.379875 (+ 592us) server_negotiation.cc:407] Connection header received
0927 10:44:14.380069 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:14.380073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:14.380122 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:14.380176 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:14.381246 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.382105 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.383502 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.383915 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.383994 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:14.388051 (+ 4057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55306: BlockingRecv error: recv got EOF from 127.0.0.1:55306 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 10:44:14 dist-test-slave-dzhw krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 10:44:14.400177 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:14.389751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55320 (local address 127.0.98.62:39745)
0927 10:44:14.389867 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:14.389871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:14.390078 (+ 207us) server_negotiation.cc:407] Connection header received
0927 10:44:14.390221 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:14.390225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:14.390276 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:14.390369 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:14.391205 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.391986 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.393134 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.393460 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.393560 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:14.396556 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:14.396560 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:14.396563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:14.396595 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:14.398320 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:14.398852 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:14.398855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:14.398857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:14.398948 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:14.399235 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:14.399238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:14.399240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:14.399468 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:14.399596 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:14.399867 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 10:44:14.399998 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:44:14.403231 7907 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:44:14.403731 7907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:44:14.409240 8186 tablet_service.cc:1467] Processing CreateTablet for tablet 480a6d30def24cdea2f8101d9d786aed (DEFAULT_TABLE table=test-table [id=aebc44088720473c965ad7d671d5cab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:14.409245 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 480a6d30def24cdea2f8101d9d786aed (DEFAULT_TABLE table=test-table [id=aebc44088720473c965ad7d671d5cab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:14.409559 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 480a6d30def24cdea2f8101d9d786aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:14.409560 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 480a6d30def24cdea2f8101d9d786aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:14.412022 8420 tablet_bootstrap.cc:492] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139: Bootstrap starting.
I20260927 10:44:14.412065 8422 tablet_bootstrap.cc:492] T 480a6d30def24cdea2f8101d9d786aed P bef3dd5461af4e37b5bace4d2f36536a: Bootstrap starting.
I20260927 10:44:14.412897 8420 tablet_bootstrap.cc:654] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:14.412977 8422 tablet_bootstrap.cc:654] T 480a6d30def24cdea2f8101d9d786aed P bef3dd5461af4e37b5bace4d2f36536a: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:14.413908 8046 tablet_service.cc:1467] Processing CreateTablet for tablet 480a6d30def24cdea2f8101d9d786aed (DEFAULT_TABLE table=test-table [id=aebc44088720473c965ad7d671d5cab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:14.414101 8420 tablet_bootstrap.cc:492] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139: No bootstrap required, opened a new log
I20260927 10:44:14.414161 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 480a6d30def24cdea2f8101d9d786aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:14.414170 8420 ts_tablet_manager.cc:1397] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:44:14.414642 8420 raft_consensus.cc:348] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.414758 8420 raft_consensus.cc:374] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:14.414789 8420 raft_consensus.cc:729] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c6b4e51bf3143a4a2bd020f699a3139, State: Initialized, Role: FOLLOWER
I20260927 10:44:14.414911 8420 consensus_queue.cc:260] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.415182 8420 ts_tablet_manager.cc:1428] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:44:14.417481 8421 tablet_bootstrap.cc:492] T 480a6d30def24cdea2f8101d9d786aed P c7de0b62beb8473a81706766c3aca2ab: Bootstrap starting.
I20260927 10:44:14.418300 8421 tablet_bootstrap.cc:654] T 480a6d30def24cdea2f8101d9d786aed P c7de0b62beb8473a81706766c3aca2ab: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:14.418378 8422 tablet_bootstrap.cc:492] T 480a6d30def24cdea2f8101d9d786aed P bef3dd5461af4e37b5bace4d2f36536a: No bootstrap required, opened a new log
I20260927 10:44:14.418494 8422 ts_tablet_manager.cc:1397] T 480a6d30def24cdea2f8101d9d786aed P bef3dd5461af4e37b5bace4d2f36536a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260927 10:44:14.418972 8422 raft_consensus.cc:348] T 480a6d30def24cdea2f8101d9d786aed P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.419085 8422 raft_consensus.cc:374] T 480a6d30def24cdea2f8101d9d786aed P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:14.419134 8422 raft_consensus.cc:729] T 480a6d30def24cdea2f8101d9d786aed P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bef3dd5461af4e37b5bace4d2f36536a, State: Initialized, Role: FOLLOWER
I20260927 10:44:14.419322 8421 tablet_bootstrap.cc:492] T 480a6d30def24cdea2f8101d9d786aed P c7de0b62beb8473a81706766c3aca2ab: No bootstrap required, opened a new log
I20260927 10:44:14.419270 8422 consensus_queue.cc:260] T 480a6d30def24cdea2f8101d9d786aed P bef3dd5461af4e37b5bace4d2f36536a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.419397 8421 ts_tablet_manager.cc:1397] T 480a6d30def24cdea2f8101d9d786aed P c7de0b62beb8473a81706766c3aca2ab: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 10:44:14.419564 8422 ts_tablet_manager.cc:1428] T 480a6d30def24cdea2f8101d9d786aed P bef3dd5461af4e37b5bace4d2f36536a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 10:44:14.419806 8421 raft_consensus.cc:348] T 480a6d30def24cdea2f8101d9d786aed P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.419909 8421 raft_consensus.cc:374] T 480a6d30def24cdea2f8101d9d786aed P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:14.419938 8421 raft_consensus.cc:729] T 480a6d30def24cdea2f8101d9d786aed P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7de0b62beb8473a81706766c3aca2ab, State: Initialized, Role: FOLLOWER
I20260927 10:44:14.420054 8421 consensus_queue.cc:260] T 480a6d30def24cdea2f8101d9d786aed P c7de0b62beb8473a81706766c3aca2ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.420306 8421 ts_tablet_manager.cc:1428] T 480a6d30def24cdea2f8101d9d786aed P c7de0b62beb8473a81706766c3aca2ab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:44:14.476147 8426 raft_consensus.cc:482] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:44:14.476336 8426 raft_consensus.cc:504] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.477458 8426 leader_election.cc:288] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7de0b62beb8473a81706766c3aca2ab (127.0.98.1:37455), bef3dd5461af4e37b5bace4d2f36536a (127.0.98.2:34385)
I20260927 10:44:14.482228 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:14.477639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37455 (local address 127.0.98.3:50671)
0927 10:44:14.477997 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:14.478013 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:14.478168 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:14.478593 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:14.478596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:14.478614 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:14.478759 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:14.478765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.479689 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.479693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:14.480820 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:14.480832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.481790 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.481794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:14.481887 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:14.481913 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 10:44:14.481986 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 10:44:14.482057 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":103,"threads_started":1}
I20260927 10:44:14.482270 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:14.477763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50671 (local address 127.0.98.1:37455)
0927 10:44:14.477997 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:14.478003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:14.478084 (+ 81us) server_negotiation.cc:407] Connection header received
0927 10:44:14.478305 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:14.478309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:14.478357 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:14.478407 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:14.478877 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.479497 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.480968 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.481604 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.481736 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:14.481807 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:14.482028 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 10:44:14.482116 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260927 10:44:14.483083 8066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "480a6d30def24cdea2f8101d9d786aed" candidate_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7de0b62beb8473a81706766c3aca2ab" is_pre_election: true
I20260927 10:44:14.483187 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:14.478205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53333 (local address 127.0.98.2:34385)
0927 10:44:14.478481 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:14.478487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:14.478568 (+ 81us) server_negotiation.cc:407] Connection header received
0927 10:44:14.478712 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:14.478716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:14.478788 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:14.478892 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:14.479162 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.480089 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.481716 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.482361 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.482508 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:14.482567 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:14.483011 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 10:44:14.483083 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 10:44:14.483201 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:14.477929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34385 (local address 127.0.98.3:53333)
0927 10:44:14.478454 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:14.478470 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:14.478581 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:14.478918 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:14.478921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:14.478941 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:14.479039 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:14.479044 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.480564 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.480567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:14.481545 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:14.481554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.482672 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.482675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:14.482886 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:14.482903 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 10:44:14.482992 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 10:44:14.483056 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":130,"threads_started":1}
I20260927 10:44:14.483255 8066 raft_consensus.cc:2454] T 480a6d30def24cdea2f8101d9d786aed P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c6b4e51bf3143a4a2bd020f699a3139 in term 0.
I20260927 10:44:14.483606 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "480a6d30def24cdea2f8101d9d786aed" candidate_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bef3dd5461af4e37b5bace4d2f36536a" is_pre_election: true
I20260927 10:44:14.483752 8206 raft_consensus.cc:2454] T 480a6d30def24cdea2f8101d9d786aed P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c6b4e51bf3143a4a2bd020f699a3139 in term 0.
I20260927 10:44:14.483686 8280 leader_election.cc:302] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c6b4e51bf3143a4a2bd020f699a3139, c7de0b62beb8473a81706766c3aca2ab; no voters:
I20260927 10:44:14.484004 8426 raft_consensus.cc:2781] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:44:14.484076 8426 raft_consensus.cc:482] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:44:14.484118 8426 raft_consensus.cc:3037] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:14.485471 8426 raft_consensus.cc:504] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.485901 8426 leader_election.cc:288] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139 [CANDIDATE]: Term 1 election: Requested vote from peers c7de0b62beb8473a81706766c3aca2ab (127.0.98.1:37455), bef3dd5461af4e37b5bace4d2f36536a (127.0.98.2:34385)
I20260927 10:44:14.486231 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "480a6d30def24cdea2f8101d9d786aed" candidate_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bef3dd5461af4e37b5bace4d2f36536a"
I20260927 10:44:14.486327 8206 raft_consensus.cc:3037] T 480a6d30def24cdea2f8101d9d786aed P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:14.486282 8066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "480a6d30def24cdea2f8101d9d786aed" candidate_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7de0b62beb8473a81706766c3aca2ab"
I20260927 10:44:14.486410 8066 raft_consensus.cc:3037] T 480a6d30def24cdea2f8101d9d786aed P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:14.487331 8206 raft_consensus.cc:2454] T 480a6d30def24cdea2f8101d9d786aed P bef3dd5461af4e37b5bace4d2f36536a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c6b4e51bf3143a4a2bd020f699a3139 in term 1.
I20260927 10:44:14.487706 8066 raft_consensus.cc:2454] T 480a6d30def24cdea2f8101d9d786aed P c7de0b62beb8473a81706766c3aca2ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c6b4e51bf3143a4a2bd020f699a3139 in term 1.
I20260927 10:44:14.487982 8278 leader_election.cc:302] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c6b4e51bf3143a4a2bd020f699a3139, bef3dd5461af4e37b5bace4d2f36536a; no voters:
I20260927 10:44:14.488161 8426 raft_consensus.cc:2781] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:14.488409 8426 raft_consensus.cc:686] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139 [term 1 LEADER]: Becoming Leader. State: Replica: 9c6b4e51bf3143a4a2bd020f699a3139, State: Running, Role: LEADER
I20260927 10:44:14.488692 8426 consensus_queue.cc:237] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.491290 7909 catalog_manager.cc:5495] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c6b4e51bf3143a4a2bd020f699a3139 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } health_report { overall_health: HEALTHY } } }
I20260927 10:44:14.512082 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:14.508158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33038 (local address 127.0.98.3:36351)
0927 10:44:14.508292 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:14.508298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:14.508347 (+ 49us) server_negotiation.cc:407] Connection header received
0927 10:44:14.508441 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:14.508444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:14.508528 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:14.508645 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:44:14.508850 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.509615 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.510477 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.510809 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.510918 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:14.511132 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:44:14.511686 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:44:14.511813 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:14.511877 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 10:44:14.511935 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:44:14.514681 8206 raft_consensus.cc:1261] T 480a6d30def24cdea2f8101d9d786aed P bef3dd5461af4e37b5bace4d2f36536a [term 1 FOLLOWER]: Refusing update from remote peer 9c6b4e51bf3143a4a2bd020f699a3139: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:14.514709 8066 raft_consensus.cc:1261] T 480a6d30def24cdea2f8101d9d786aed P c7de0b62beb8473a81706766c3aca2ab [term 1 FOLLOWER]: Refusing update from remote peer 9c6b4e51bf3143a4a2bd020f699a3139: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:14.515210 8497 consensus_queue.cc:1035] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:14.515465 8426 consensus_queue.cc:1035] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:14.528226 7909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:14.528388 7909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:14.530453 7909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3: Sending DeleteTablet for 3 replicas of tablet 480a6d30def24cdea2f8101d9d786aed
I20260927 10:44:14.530999 8046 tablet_service.cc:1514] Processing DeleteTablet for tablet 480a6d30def24cdea2f8101d9d786aed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50200
I20260927 10:44:14.531095 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet 480a6d30def24cdea2f8101d9d786aed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40530
I20260927 10:44:14.531229 8445 tablet_replica.cc:330] T 480a6d30def24cdea2f8101d9d786aed P c7de0b62beb8473a81706766c3aca2ab: stopping tablet replica
I20260927 10:44:14.531308 8444 tablet_replica.cc:330] T 480a6d30def24cdea2f8101d9d786aed P bef3dd5461af4e37b5bace4d2f36536a: stopping tablet replica
I20260927 10:44:14.531350 8445 raft_consensus.cc:2229] T 480a6d30def24cdea2f8101d9d786aed P c7de0b62beb8473a81706766c3aca2ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:14.531423 8444 raft_consensus.cc:2229] T 480a6d30def24cdea2f8101d9d786aed P bef3dd5461af4e37b5bace4d2f36536a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:14.531494 8445 raft_consensus.cc:2258] T 480a6d30def24cdea2f8101d9d786aed P c7de0b62beb8473a81706766c3aca2ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:14.531558 8444 raft_consensus.cc:2258] T 480a6d30def24cdea2f8101d9d786aed P bef3dd5461af4e37b5bace4d2f36536a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:14.531669 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet 480a6d30def24cdea2f8101d9d786aed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37592
I20260927 10:44:14.531855 8446 tablet_replica.cc:330] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139: stopping tablet replica
I20260927 10:44:14.531977 8446 raft_consensus.cc:2229] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:44:14.532310 8446 raft_consensus.cc:2258] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:14.532471 8444 ts_tablet_manager.cc:1905] T 480a6d30def24cdea2f8101d9d786aed P bef3dd5461af4e37b5bace4d2f36536a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:14.532537 8445 ts_tablet_manager.cc:1905] T 480a6d30def24cdea2f8101d9d786aed P c7de0b62beb8473a81706766c3aca2ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:14.532815 7909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:44:14.533195 7909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:44:14.535032 8444 ts_tablet_manager.cc:1918] T 480a6d30def24cdea2f8101d9d786aed P bef3dd5461af4e37b5bace4d2f36536a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:44:14.535123 8444 log.cc:1196] T 480a6d30def24cdea2f8101d9d786aed P bef3dd5461af4e37b5bace4d2f36536a: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-1/wal/wals/480a6d30def24cdea2f8101d9d786aed
I20260927 10:44:14.535120 8445 ts_tablet_manager.cc:1918] T 480a6d30def24cdea2f8101d9d786aed P c7de0b62beb8473a81706766c3aca2ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:44:14.535188 8445 log.cc:1196] T 480a6d30def24cdea2f8101d9d786aed P c7de0b62beb8473a81706766c3aca2ab: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-0/wal/wals/480a6d30def24cdea2f8101d9d786aed
I20260927 10:44:14.535393 8444 ts_tablet_manager.cc:1939] T 480a6d30def24cdea2f8101d9d786aed P bef3dd5461af4e37b5bace4d2f36536a: Deleting consensus metadata
I20260927 10:44:14.535444 8445 ts_tablet_manager.cc:1939] T 480a6d30def24cdea2f8101d9d786aed P c7de0b62beb8473a81706766c3aca2ab: Deleting consensus metadata
I20260927 10:44:14.536041 8446 ts_tablet_manager.cc:1905] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:14.536182 7896 catalog_manager.cc:4841] TS c7de0b62beb8473a81706766c3aca2ab (127.0.98.1:37455): tablet 480a6d30def24cdea2f8101d9d786aed (table test-table [id=aebc44088720473c965ad7d671d5cab7]) successfully deleted
I20260927 10:44:14.536572 7894 catalog_manager.cc:4841] TS bef3dd5461af4e37b5bace4d2f36536a (127.0.98.2:34385): tablet 480a6d30def24cdea2f8101d9d786aed (table test-table [id=aebc44088720473c965ad7d671d5cab7]) successfully deleted
I20260927 10:44:14.538385 8446 ts_tablet_manager.cc:1918] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:44:14.538484 8446 log.cc:1196] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/480a6d30def24cdea2f8101d9d786aed
I20260927 10:44:14.538738 8446 ts_tablet_manager.cc:1939] T 480a6d30def24cdea2f8101d9d786aed P 9c6b4e51bf3143a4a2bd020f699a3139: Deleting consensus metadata
I20260927 10:44:14.539315 8186 tablet_service.cc:1467] Processing CreateTablet for tablet d81c7147bf54479ab0dc81a17536f2db (DEFAULT_TABLE table=test-table [id=22eb683a2bff403b8ffc73817baf7241]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:14.539600 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81c7147bf54479ab0dc81a17536f2db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:14.540076 7896 catalog_manager.cc:4841] TS 9c6b4e51bf3143a4a2bd020f699a3139 (127.0.98.3:36351): tablet 480a6d30def24cdea2f8101d9d786aed (table test-table [id=aebc44088720473c965ad7d671d5cab7]) successfully deleted
I20260927 10:44:14.540170 8309 tablet_service.cc:1467] Processing CreateTablet for tablet d81c7147bf54479ab0dc81a17536f2db (DEFAULT_TABLE table=test-table [id=22eb683a2bff403b8ffc73817baf7241]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:14.540381 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81c7147bf54479ab0dc81a17536f2db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:14.540570 8046 tablet_service.cc:1467] Processing CreateTablet for tablet d81c7147bf54479ab0dc81a17536f2db (DEFAULT_TABLE table=test-table [id=22eb683a2bff403b8ffc73817baf7241]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:14.540813 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81c7147bf54479ab0dc81a17536f2db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:14.541904 8422 tablet_bootstrap.cc:492] T d81c7147bf54479ab0dc81a17536f2db P bef3dd5461af4e37b5bace4d2f36536a: Bootstrap starting.
I20260927 10:44:14.542613 8420 tablet_bootstrap.cc:492] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139: Bootstrap starting.
I20260927 10:44:14.542975 8422 tablet_bootstrap.cc:654] T d81c7147bf54479ab0dc81a17536f2db P bef3dd5461af4e37b5bace4d2f36536a: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:14.543004 8421 tablet_bootstrap.cc:492] T d81c7147bf54479ab0dc81a17536f2db P c7de0b62beb8473a81706766c3aca2ab: Bootstrap starting.
I20260927 10:44:14.543553 8420 tablet_bootstrap.cc:654] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:14.543733 8421 tablet_bootstrap.cc:654] T d81c7147bf54479ab0dc81a17536f2db P c7de0b62beb8473a81706766c3aca2ab: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:14.544562 8420 tablet_bootstrap.cc:492] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139: No bootstrap required, opened a new log
I20260927 10:44:14.544613 8422 tablet_bootstrap.cc:492] T d81c7147bf54479ab0dc81a17536f2db P bef3dd5461af4e37b5bace4d2f36536a: No bootstrap required, opened a new log
I20260927 10:44:14.544651 8420 ts_tablet_manager.cc:1397] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:44:14.544677 8422 ts_tablet_manager.cc:1397] T d81c7147bf54479ab0dc81a17536f2db P bef3dd5461af4e37b5bace4d2f36536a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 10:44:14.545044 8422 raft_consensus.cc:348] T d81c7147bf54479ab0dc81a17536f2db P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.545069 8420 raft_consensus.cc:348] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.545172 8422 raft_consensus.cc:374] T d81c7147bf54479ab0dc81a17536f2db P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:14.545172 8420 raft_consensus.cc:374] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:14.545207 8420 raft_consensus.cc:729] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c6b4e51bf3143a4a2bd020f699a3139, State: Initialized, Role: FOLLOWER
I20260927 10:44:14.545207 8422 raft_consensus.cc:729] T d81c7147bf54479ab0dc81a17536f2db P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bef3dd5461af4e37b5bace4d2f36536a, State: Initialized, Role: FOLLOWER
I20260927 10:44:14.545325 8420 consensus_queue.cc:260] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.545324 8422 consensus_queue.cc:260] T d81c7147bf54479ab0dc81a17536f2db P bef3dd5461af4e37b5bace4d2f36536a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.545583 8422 ts_tablet_manager.cc:1428] T d81c7147bf54479ab0dc81a17536f2db P bef3dd5461af4e37b5bace4d2f36536a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:44:14.545609 8420 ts_tablet_manager.cc:1428] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:44:14.546152 8421 tablet_bootstrap.cc:492] T d81c7147bf54479ab0dc81a17536f2db P c7de0b62beb8473a81706766c3aca2ab: No bootstrap required, opened a new log
I20260927 10:44:14.546227 8421 ts_tablet_manager.cc:1397] T d81c7147bf54479ab0dc81a17536f2db P c7de0b62beb8473a81706766c3aca2ab: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:44:14.546633 8421 raft_consensus.cc:348] T d81c7147bf54479ab0dc81a17536f2db P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.546757 8421 raft_consensus.cc:374] T d81c7147bf54479ab0dc81a17536f2db P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:14.546788 8421 raft_consensus.cc:729] T d81c7147bf54479ab0dc81a17536f2db P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7de0b62beb8473a81706766c3aca2ab, State: Initialized, Role: FOLLOWER
I20260927 10:44:14.546900 8421 consensus_queue.cc:260] T d81c7147bf54479ab0dc81a17536f2db P c7de0b62beb8473a81706766c3aca2ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.547159 8421 ts_tablet_manager.cc:1428] T d81c7147bf54479ab0dc81a17536f2db P c7de0b62beb8473a81706766c3aca2ab: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:44:14.748713 8497 raft_consensus.cc:482] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:44:14.748864 8497 raft_consensus.cc:504] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.749415 8497 leader_election.cc:288] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7de0b62beb8473a81706766c3aca2ab (127.0.98.1:37455), bef3dd5461af4e37b5bace4d2f36536a (127.0.98.2:34385)
I20260927 10:44:14.750056 8066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81c7147bf54479ab0dc81a17536f2db" candidate_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7de0b62beb8473a81706766c3aca2ab" is_pre_election: true
I20260927 10:44:14.750056 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81c7147bf54479ab0dc81a17536f2db" candidate_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bef3dd5461af4e37b5bace4d2f36536a" is_pre_election: true
I20260927 10:44:14.750211 8066 raft_consensus.cc:2454] T d81c7147bf54479ab0dc81a17536f2db P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c6b4e51bf3143a4a2bd020f699a3139 in term 0.
I20260927 10:44:14.750240 8206 raft_consensus.cc:2454] T d81c7147bf54479ab0dc81a17536f2db P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c6b4e51bf3143a4a2bd020f699a3139 in term 0.
I20260927 10:44:14.750545 8278 leader_election.cc:302] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c6b4e51bf3143a4a2bd020f699a3139, bef3dd5461af4e37b5bace4d2f36536a; no voters:
I20260927 10:44:14.750730 8497 raft_consensus.cc:2781] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:44:14.750788 8497 raft_consensus.cc:482] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:44:14.750830 8497 raft_consensus.cc:3037] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:14.751865 8497 raft_consensus.cc:504] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.752143 8497 leader_election.cc:288] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139 [CANDIDATE]: Term 1 election: Requested vote from peers c7de0b62beb8473a81706766c3aca2ab (127.0.98.1:37455), bef3dd5461af4e37b5bace4d2f36536a (127.0.98.2:34385)
I20260927 10:44:14.752489 8205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81c7147bf54479ab0dc81a17536f2db" candidate_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bef3dd5461af4e37b5bace4d2f36536a"
I20260927 10:44:14.752491 8066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81c7147bf54479ab0dc81a17536f2db" candidate_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7de0b62beb8473a81706766c3aca2ab"
I20260927 10:44:14.752595 8205 raft_consensus.cc:3037] T d81c7147bf54479ab0dc81a17536f2db P bef3dd5461af4e37b5bace4d2f36536a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:14.752625 8066 raft_consensus.cc:3037] T d81c7147bf54479ab0dc81a17536f2db P c7de0b62beb8473a81706766c3aca2ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:14.753674 8066 raft_consensus.cc:2454] T d81c7147bf54479ab0dc81a17536f2db P c7de0b62beb8473a81706766c3aca2ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c6b4e51bf3143a4a2bd020f699a3139 in term 1.
I20260927 10:44:14.753697 8205 raft_consensus.cc:2454] T d81c7147bf54479ab0dc81a17536f2db P bef3dd5461af4e37b5bace4d2f36536a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c6b4e51bf3143a4a2bd020f699a3139 in term 1.
I20260927 10:44:14.753997 8280 leader_election.cc:302] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c6b4e51bf3143a4a2bd020f699a3139, c7de0b62beb8473a81706766c3aca2ab; no voters:
I20260927 10:44:14.754199 8497 raft_consensus.cc:2781] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:14.754309 8497 raft_consensus.cc:686] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139 [term 1 LEADER]: Becoming Leader. State: Replica: 9c6b4e51bf3143a4a2bd020f699a3139, State: Running, Role: LEADER
I20260927 10:44:14.754477 8497 consensus_queue.cc:237] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } }
I20260927 10:44:14.756116 7909 catalog_manager.cc:5495] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c6b4e51bf3143a4a2bd020f699a3139 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c6b4e51bf3143a4a2bd020f699a3139" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36351 } health_report { overall_health: HEALTHY } } }
I20260927 10:44:14.798086 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:14.795194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53505 (local address 127.0.98.1:37455)
0927 10:44:14.795323 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:14.795327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:14.795511 (+ 184us) server_negotiation.cc:407] Connection header received
0927 10:44:14.795619 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:14.795621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:14.795664 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:14.795737 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:44:14.795931 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.796419 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.797204 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.797418 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.797499 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:14.797743 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:44:14.797809 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:44:14.797868 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:14.797939 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 10:44:14.797978 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:44:14.798101 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:14.795064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37455 (local address 127.0.98.3:53505)
0927 10:44:14.795437 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:14.795449 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:14.795537 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:14.795752 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:14.795754 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:14.795760 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:44:14.795847 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:14.795851 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.796503 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.796505 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:14.797099 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:14.797106 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.797514 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.797517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:14.797572 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:14.797611 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:44:14.797881 (+ 270us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:44:14.797885 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 10:44:14.797931 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 10:44:14.797992 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":120,"threads_started":1}
I20260927 10:44:14.803584 8205 raft_consensus.cc:1261] T d81c7147bf54479ab0dc81a17536f2db P bef3dd5461af4e37b5bace4d2f36536a [term 1 FOLLOWER]: Refusing update from remote peer 9c6b4e51bf3143a4a2bd020f699a3139: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:14.803632 8066 raft_consensus.cc:1261] T d81c7147bf54479ab0dc81a17536f2db P c7de0b62beb8473a81706766c3aca2ab [term 1 FOLLOWER]: Refusing update from remote peer 9c6b4e51bf3143a4a2bd020f699a3139: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:14.803953 8502 consensus_queue.cc:1035] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bef3dd5461af4e37b5bace4d2f36536a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:14.804145 8497 consensus_queue.cc:1035] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7de0b62beb8473a81706766c3aca2ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:14.821341 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:14.817992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36351 (local address 127.0.98.1:39579)
0927 10:44:14.818234 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:14.818250 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:14.818349 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:14.818592 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:14.818594 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:14.818600 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:44:14.818693 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:14.818697 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.819365 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.819368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:14.820056 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:14.820065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.820622 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.820626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:14.820712 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:14.820749 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:44:14.821062 (+ 313us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:44:14.821069 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 10:44:14.821144 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 10:44:14.821221 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":73,"threads_started":1}
I20260927 10:44:14.821336 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:14.818064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39579 (local address 127.0.98.3:36351)
0927 10:44:14.818185 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:14.818191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:14.818302 (+ 111us) server_negotiation.cc:407] Connection header received
0927 10:44:14.818468 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:14.818471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:14.818523 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:14.818581 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:44:14.818763 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.819250 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.820166 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:14.820522 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:14.820624 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:14.820863 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:44:14.820945 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:44:14.821014 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:14.821178 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 10:44:14.821240 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:44:14.840090 7907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:14.840238 7907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:14.842110 7907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ae783e79d7b4186b372344edaec95d3: Sending DeleteTablet for 3 replicas of tablet d81c7147bf54479ab0dc81a17536f2db
I20260927 10:44:14.842628 8046 tablet_service.cc:1514] Processing DeleteTablet for tablet d81c7147bf54479ab0dc81a17536f2db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50200
I20260927 10:44:14.842749 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet d81c7147bf54479ab0dc81a17536f2db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40530
I20260927 10:44:14.842847 8445 tablet_replica.cc:330] T d81c7147bf54479ab0dc81a17536f2db P c7de0b62beb8473a81706766c3aca2ab: stopping tablet replica
I20260927 10:44:14.842945 8444 tablet_replica.cc:330] T d81c7147bf54479ab0dc81a17536f2db P bef3dd5461af4e37b5bace4d2f36536a: stopping tablet replica
I20260927 10:44:14.842967 8445 raft_consensus.cc:2229] T d81c7147bf54479ab0dc81a17536f2db P c7de0b62beb8473a81706766c3aca2ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:14.843082 8444 raft_consensus.cc:2229] T d81c7147bf54479ab0dc81a17536f2db P bef3dd5461af4e37b5bace4d2f36536a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:14.843103 8445 raft_consensus.cc:2258] T d81c7147bf54479ab0dc81a17536f2db P c7de0b62beb8473a81706766c3aca2ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:14.843219 8444 raft_consensus.cc:2258] T d81c7147bf54479ab0dc81a17536f2db P bef3dd5461af4e37b5bace4d2f36536a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:14.843485 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet d81c7147bf54479ab0dc81a17536f2db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37592
I20260927 10:44:14.843669 8446 tablet_replica.cc:330] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139: stopping tablet replica
I20260927 10:44:14.843783 8446 raft_consensus.cc:2229] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:44:14.843808 8445 ts_tablet_manager.cc:1905] T d81c7147bf54479ab0dc81a17536f2db P c7de0b62beb8473a81706766c3aca2ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:14.843976 8446 raft_consensus.cc:2258] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:14.844512 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 7981
I20260927 10:44:14.844712 8446 ts_tablet_manager.cc:1905] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:14.847375 8446 ts_tablet_manager.cc:1918] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:44:14.847462 8446 log.cc:1196] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/d81c7147bf54479ab0dc81a17536f2db
I20260927 10:44:14.847716 8446 ts_tablet_manager.cc:1939] T d81c7147bf54479ab0dc81a17536f2db P 9c6b4e51bf3143a4a2bd020f699a3139: Deleting consensus metadata
I20260927 10:44:14.848136 8444 ts_tablet_manager.cc:1905] T d81c7147bf54479ab0dc81a17536f2db P bef3dd5461af4e37b5bace4d2f36536a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:14.848381 7896 catalog_manager.cc:4841] TS 9c6b4e51bf3143a4a2bd020f699a3139 (127.0.98.3:36351): tablet d81c7147bf54479ab0dc81a17536f2db (table test-table [id=22eb683a2bff403b8ffc73817baf7241]) successfully deleted
I20260927 10:44:14.850059 8444 ts_tablet_manager.cc:1918] T d81c7147bf54479ab0dc81a17536f2db P bef3dd5461af4e37b5bace4d2f36536a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:44:14.850134 8444 log.cc:1196] T d81c7147bf54479ab0dc81a17536f2db P bef3dd5461af4e37b5bace4d2f36536a: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505812134190-392-0/minicluster-data/ts-1/wal/wals/d81c7147bf54479ab0dc81a17536f2db
I20260927 10:44:14.850366 8444 ts_tablet_manager.cc:1939] T d81c7147bf54479ab0dc81a17536f2db P bef3dd5461af4e37b5bace4d2f36536a: Deleting consensus metadata
I20260927 10:44:14.851052 7894 catalog_manager.cc:4841] TS bef3dd5461af4e37b5bace4d2f36536a (127.0.98.2:34385): tablet d81c7147bf54479ab0dc81a17536f2db (table test-table [id=22eb683a2bff403b8ffc73817baf7241]) successfully deleted
W20260927 10:44:14.854513 7896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37455 (error 108)
I20260927 10:44:14.854655 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 8121
I20260927 10:44:14.856900 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:14.856457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37455 (local address 127.0.0.1:37556)
0927 10:44:14.856687 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:14.856770 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":51,"threads_started":1}
W20260927 10:44:14.857056 7896 catalog_manager.cc:4584] TS c7de0b62beb8473a81706766c3aca2ab (127.0.98.1:37455): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d81c7147bf54479ab0dc81a17536f2db: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37455: connect: Connection refused (error 111)
I20260927 10:44:14.861876 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 8261
I20260927 10:44:14.868669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 7877
2026-09-27T10:44:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3364 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:44:14 dist-test-slave-dzhw krb5kdc[8520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:44:14 dist-test-slave-dzhw krb5kdc[8520](info): set up 2 sockets
Sep 27 10:44:14 dist-test-slave-dzhw krb5kdc[8520](info): commencing operation
krb5kdc: starting...
W20260927 10:44:15.919785 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:44:16 dist-test-slave-dzhw krb5kdc[8520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:44:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:44:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:44:16.046583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44405
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:16.156430 8533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:16.156711 8533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:16.156752 8533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:16.160815 8533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:44:16.160882 8533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:16.160902 8533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:16.160918 8533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:44:16.160934 8533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:44:16.166215 8533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:16.167407 8533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:16.168435 8533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:44:16.176999 8533 server_base.cc:1034] running on GCE node
W20260927 10:44:16.176951 8541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:16.176992 8538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:16.176951 8539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:16.177801 8533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:16.179198 8533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:16.180380 8533 hybrid_clock.cc:648] HybridClock initialized: now 1790505856180352 us; error 44 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T10:44:16Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1324 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:44:16 dist-test-slave-dzhw krb5kdc[8546](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:44:16 dist-test-slave-dzhw krb5kdc[8546](info): set up 2 sockets
Sep 27 10:44:16 dist-test-slave-dzhw krb5kdc[8546](info): commencing operation
krb5kdc: starting...
W20260927 10:44:17.239773 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:44:17 dist-test-slave-dzhw krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:44:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:44:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:44:17.366705 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36955
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:17.476099 8559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:17.476433 8559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:17.476506 8559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:17.480563 8559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:44:17.480670 8559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:17.480708 8559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:17.480770 8559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:44:17.480827 8559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:44:17.486251 8559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:17.487426 8559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:17.488461 8559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:17.497123 8564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:17.497638 8567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:17.497627 8565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:17.497622 8559 server_base.cc:1034] running on GCE node
I20260927 10:44:17.498577 8559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:17.499732 8559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:17.500936 8559 hybrid_clock.cc:648] HybridClock initialized: now 1790505857500921 us; error 36 us; skew 500 ppm
Sep 27 10:44:17 dist-test-slave-dzhw krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:17.503891 8559 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:44:17.504792 8559 webserver.cc:460] Webserver started at http://127.0.98.62:39103/ using document root <none> and password file <none>
I20260927 10:44:17.505375 8559 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:17.505466 8559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:17.505697 8559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:17.507864 8559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "395d5988443d41fb9d078e20ad0a2366"
format_stamp: "Formatted at 2026-09-27 10:44:17 on dist-test-slave-dzhw"
server_key: "51f534e09ace45f8ce35581c1be1a520"
server_key_iv: "dcee0d00c9a0d9cfde524230ac77e19c"
server_key_version: "encryptionkey@0"
I20260927 10:44:17.508442 8559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "395d5988443d41fb9d078e20ad0a2366"
format_stamp: "Formatted at 2026-09-27 10:44:17 on dist-test-slave-dzhw"
server_key: "51f534e09ace45f8ce35581c1be1a520"
server_key_iv: "dcee0d00c9a0d9cfde524230ac77e19c"
server_key_version: "encryptionkey@0"
I20260927 10:44:17.512804 8559 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:44:17.515575 8574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:17.516959 8559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:44:17.517130 8559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "395d5988443d41fb9d078e20ad0a2366"
format_stamp: "Formatted at 2026-09-27 10:44:17 on dist-test-slave-dzhw"
server_key: "51f534e09ace45f8ce35581c1be1a520"
server_key_iv: "dcee0d00c9a0d9cfde524230ac77e19c"
server_key_version: "encryptionkey@0"
I20260927 10:44:17.517256 8559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:17.538645 8559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:17.541782 8559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:17.542014 8559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:17.549201 8559 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36955
I20260927 10:44:17.549216 8626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36955 every 8 connection(s)
I20260927 10:44:17.550201 8559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:44:17.553171 8627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:17.553241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 8559
I20260927 10:44:17.553354 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:44:17.553663 392 external_mini_cluster.cc:1428] Setting key 7bdf1ecab0e46fd2e41f723631cb8f0a
I20260927 10:44:17.560045 8627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366: Bootstrap starting.
Sep 27 10:44:17 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:44:17.562867 8627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:17.563709 8627 log.cc:824] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:17.566256 8627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366: No bootstrap required, opened a new log
I20260927 10:44:17.568902 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:17.555093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50438 (local address 127.0.98.62:36955)
0927 10:44:17.555512 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:17.555527 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:17.555562 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:44:17.556337 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:17.556356 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:17.556684 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:17.556965 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:17.557840 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:17.558902 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:17.560027 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:17.560556 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:17.560684 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:17.563441 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:17.563447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:17.563463 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:17.563499 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:17.565650 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:17.566090 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:17.566095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:17.566101 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:17.566233 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:17.566613 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:17.566616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:17.566618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:17.567138 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:17.567325 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:17.567533 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 10:44:17.567859 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":137,"threads_started":1}
I20260927 10:44:17.570751 8627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395d5988443d41fb9d078e20ad0a2366" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36955 } }
I20260927 10:44:17.571018 8627 raft_consensus.cc:374] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:17.571089 8627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395d5988443d41fb9d078e20ad0a2366, State: Initialized, Role: FOLLOWER
I20260927 10:44:17.571554 8627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395d5988443d41fb9d078e20ad0a2366" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36955 } }
I20260927 10:44:17.571704 8627 raft_consensus.cc:388] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:44:17.571772 8627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:44:17.571869 8627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:17.572780 8627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395d5988443d41fb9d078e20ad0a2366" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36955 } }
I20260927 10:44:17.573110 8627 leader_election.cc:302] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 395d5988443d41fb9d078e20ad0a2366; no voters:
I20260927 10:44:17.573424 8627 leader_election.cc:288] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:44:17.573583 8632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:17.573841 8632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [term 1 LEADER]: Becoming Leader. State: Replica: 395d5988443d41fb9d078e20ad0a2366, State: Running, Role: LEADER
I20260927 10:44:17.574162 8632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395d5988443d41fb9d078e20ad0a2366" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36955 } }
I20260927 10:44:17.574728 8627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:44:17.575845 8633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "395d5988443d41fb9d078e20ad0a2366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395d5988443d41fb9d078e20ad0a2366" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36955 } } }
I20260927 10:44:17.576053 8633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:17.576181 8634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 395d5988443d41fb9d078e20ad0a2366. Latest consensus state: current_term: 1 leader_uuid: "395d5988443d41fb9d078e20ad0a2366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395d5988443d41fb9d078e20ad0a2366" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36955 } } }
I20260927 10:44:17.576282 8634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:17.576505 8641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:44:17.579203 8641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:44:17.584879 8641 catalog_manager.cc:1347] Generated new cluster ID: 4bc9e34dd59a4dc88a38b18fc067088d
I20260927 10:44:17.584970 8641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:44:17.592625 8641 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:44:17.593638 8641 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:44:17.602933 8641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366: Generated new TSK 0
I20260927 10:44:17.603705 8641 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 10:44:17.634327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36955
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:17.748517 8653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:17.748808 8653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:17.748852 8653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:17.753121 8653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:17.753203 8653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:17.753265 8653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:44:17.758837 8653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:17.760047 8653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:17.761103 8653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:17.771130 8661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:17.771118 8658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:17.771180 8653 server_base.cc:1034] running on GCE node
W20260927 10:44:17.771122 8659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:17.771771 8653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:17.772460 8653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:17.773643 8653 hybrid_clock.cc:648] HybridClock initialized: now 1790505857773608 us; error 46 us; skew 500 ppm
Sep 27 10:44:17 dist-test-slave-dzhw krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:17.776656 8653 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 10:44:17.777602 8653 webserver.cc:460] Webserver started at http://127.0.98.1:40815/ using document root <none> and password file <none>
I20260927 10:44:17.778223 8653 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:17.778317 8653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:17.778597 8653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:17.780938 8653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "5a4d74ef143d40ddbdab2db64ff5a9ad"
format_stamp: "Formatted at 2026-09-27 10:44:17 on dist-test-slave-dzhw"
server_key: "955159a1a2d1bc9957c9460185e26322"
server_key_iv: "b72e9b58c135e4eba8c221ba7fa2f60b"
server_key_version: "encryptionkey@0"
I20260927 10:44:17.781531 8653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5a4d74ef143d40ddbdab2db64ff5a9ad"
format_stamp: "Formatted at 2026-09-27 10:44:17 on dist-test-slave-dzhw"
server_key: "955159a1a2d1bc9957c9460185e26322"
server_key_iv: "b72e9b58c135e4eba8c221ba7fa2f60b"
server_key_version: "encryptionkey@0"
I20260927 10:44:17.785363 8653 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 10:44:17.788106 8668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:17.789523 8653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 10:44:17.789688 8653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "5a4d74ef143d40ddbdab2db64ff5a9ad"
format_stamp: "Formatted at 2026-09-27 10:44:17 on dist-test-slave-dzhw"
server_key: "955159a1a2d1bc9957c9460185e26322"
server_key_iv: "b72e9b58c135e4eba8c221ba7fa2f60b"
server_key_version: "encryptionkey@0"
I20260927 10:44:17.789819 8653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:17.805603 8653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:17.808753 8653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:17.808969 8653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:17.809759 8653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:17.810940 8653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:17.811005 8653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:17.811069 8653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:17.811098 8653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:17.820452 8653 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36753
I20260927 10:44:17.820508 8781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36753 every 8 connection(s)
I20260927 10:44:17.821497 8653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 10:44:17 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:44:17.831343 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 8653
I20260927 10:44:17.831467 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:44:17.831844 392 external_mini_cluster.cc:1428] Setting key bf7b738b88fb96b37de36c2bafc84908
I20260927 10:44:17.835410 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:17.823393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52325 (local address 127.0.98.62:36955)
0927 10:44:17.823522 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:17.823526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:17.824329 (+ 803us) server_negotiation.cc:407] Connection header received
0927 10:44:17.825731 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:17.825735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:17.825793 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:17.825871 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:17.827121 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:17.827660 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:17.828771 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:17.829015 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:17.829133 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:17.831631 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:17.831636 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:17.831640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:17.831672 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:17.833186 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:17.833693 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:17.833698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:17.833700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:17.833796 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:17.834108 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:17.834115 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:17.834120 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:17.834382 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:17.834505 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:17.835122 (+ 617us) server_negotiation.cc:299] Negotiation successful
0927 10:44:17.835248 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:44:17.836423 8784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:17.823729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.1:52325)
0927 10:44:17.824177 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:17.824220 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:17.825497 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:17.826060 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:17.826067 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:17.826488 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:17.826959 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:17.826972 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:17.827787 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:17.827791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:17.828644 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:17.828654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:17.829181 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:17.829185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:17.829346 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:17.829890 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:17.829905 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:17.831387 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:17.833299 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:17.833306 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:17.833323 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:17.833591 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:17.833886 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:17.833888 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:17.833890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:17.834019 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:17.834582 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:17.834587 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:17.834869 (+ 282us) client_negotiation.cc:769] Sending connection context
0927 10:44:17.835079 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 10:44:17.835436 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":107,"threads_started":1}
I20260927 10:44:17.837668 8782 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36955
I20260927 10:44:17.837927 8782 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:17.838567 8782 heartbeater.cc:511] Master 127.0.98.62:36955 requested a full tablet report, sending...
I20260927 10:44:17.840142 8590 ts_manager.cc:194] Registered new tserver with Master: 5a4d74ef143d40ddbdab2db64ff5a9ad (127.0.98.1:36753)
I20260927 10:44:17.841229 8590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52325
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 10:44:17.861474 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36955
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:17.973163 8787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:17.973428 8787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:17.973470 8787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:17.977803 8787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:17.977864 8787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:17.977923 8787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:44:17.983081 8787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:17.984063 8787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:17.984987 8787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:17.994778 8792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:17.994774 8795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:17.994822 8793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:17.995039 8787 server_base.cc:1034] running on GCE node
I20260927 10:44:17.995513 8787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:17.996148 8787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:17.997351 8787 hybrid_clock.cc:648] HybridClock initialized: now 1790505857997315 us; error 49 us; skew 500 ppm
Sep 27 10:44:17 dist-test-slave-dzhw krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:18.000207 8787 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 10:44:18.001127 8787 webserver.cc:460] Webserver started at http://127.0.98.2:34655/ using document root <none> and password file <none>
I20260927 10:44:18.001693 8787 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:18.001785 8787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:18.002015 8787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:18.004128 8787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "0e9d0937a32f4426a4e935bebf0054e0"
format_stamp: "Formatted at 2026-09-27 10:44:17 on dist-test-slave-dzhw"
server_key: "11c55b12cb38d43edc904a7779e279dc"
server_key_iv: "96f508cdef0472d9a055239703e29c29"
server_key_version: "encryptionkey@0"
I20260927 10:44:18.004693 8787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0e9d0937a32f4426a4e935bebf0054e0"
format_stamp: "Formatted at 2026-09-27 10:44:17 on dist-test-slave-dzhw"
server_key: "11c55b12cb38d43edc904a7779e279dc"
server_key_iv: "96f508cdef0472d9a055239703e29c29"
server_key_version: "encryptionkey@0"
I20260927 10:44:18.008657 8787 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 10:44:18.011317 8802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:18.012517 8787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:44:18.012658 8787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "0e9d0937a32f4426a4e935bebf0054e0"
format_stamp: "Formatted at 2026-09-27 10:44:17 on dist-test-slave-dzhw"
server_key: "11c55b12cb38d43edc904a7779e279dc"
server_key_iv: "96f508cdef0472d9a055239703e29c29"
server_key_version: "encryptionkey@0"
I20260927 10:44:18.012777 8787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:18.045568 8787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:18.048916 8787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:18.049132 8787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:18.049733 8787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:18.050611 8787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:18.050678 8787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:18.050745 8787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:18.050796 8787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:18.059652 8787 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34865
I20260927 10:44:18.059669 8915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34865 every 8 connection(s)
I20260927 10:44:18.060561 8787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:44:18.068301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 8787
I20260927 10:44:18.068420 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:44:18.068686 392 external_mini_cluster.cc:1428] Setting key 3bef7138e112fe14f6ba605d53c853f6
Sep 27 10:44:18 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:44:18.074744 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:18.062503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34825 (local address 127.0.98.62:36955)
0927 10:44:18.062624 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:18.062628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:18.063435 (+ 807us) server_negotiation.cc:407] Connection header received
0927 10:44:18.064838 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:18.064841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:18.064899 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:18.065001 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:18.066132 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.066721 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.067853 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.068106 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.068214 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:18.070978 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:18.070984 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:18.070990 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:18.071023 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:18.072647 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:18.073158 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:18.073162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:18.073164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:18.073252 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:18.073597 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:18.073600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:18.073602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:18.073836 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:18.073931 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:18.074457 (+ 526us) server_negotiation.cc:299] Negotiation successful
0927 10:44:18.074575 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 10:44:18.075719 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:18.062859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.2:34825)
0927 10:44:18.063289 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:18.063332 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:18.064610 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:18.065172 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:18.065179 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:18.065529 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:18.065991 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:18.066003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.066845 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.066848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:18.067716 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:18.067735 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.068260 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.068263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:18.068566 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:18.069231 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:18.069246 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:18.070819 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:18.072772 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:18.072778 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:18.072794 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:18.073050 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:18.073345 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:18.073350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:18.073352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:18.073493 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:18.073928 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:18.073935 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:18.074189 (+ 254us) client_negotiation.cc:769] Sending connection context
0927 10:44:18.074380 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 10:44:18.074736 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":104,"threads_started":1}
I20260927 10:44:18.077251 8916 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36955
I20260927 10:44:18.077503 8916 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:18.078038 8916 heartbeater.cc:511] Master 127.0.98.62:36955 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 10:44:18.079128 8590 ts_manager.cc:194] Registered new tserver with Master: 0e9d0937a32f4426a4e935bebf0054e0 (127.0.98.2:34865)
I20260927 10:44:18.079734 8590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 10:44:18.096964 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36955
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:18.215862 8921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:18.216137 8921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:18.216189 8921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:18.220407 8921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:18.220477 8921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:18.220538 8921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:44:18.225893 8921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:18.227039 8921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:18.228003 8921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:18.237461 8927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:18.237450 8926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:18.237499 8921 server_base.cc:1034] running on GCE node
W20260927 10:44:18.237432 8929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:18.238083 8921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:18.238849 8921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:18.240041 8921 hybrid_clock.cc:648] HybridClock initialized: now 1790505858240030 us; error 52 us; skew 500 ppm
Sep 27 10:44:18 dist-test-slave-dzhw krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:18.242970 8921 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 10:44:18.243882 8921 webserver.cc:460] Webserver started at http://127.0.98.3:34053/ using document root <none> and password file <none>
I20260927 10:44:18.244421 8921 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:18.244480 8921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:18.244654 8921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:18.246739 8921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "59cc9eb99c984b3399dde1c2e4a0220f"
format_stamp: "Formatted at 2026-09-27 10:44:18 on dist-test-slave-dzhw"
server_key: "5a161aa0eafc1767f2b0336923c3b670"
server_key_iv: "e0b409951d1b80aaa5488ea2fda6cc03"
server_key_version: "encryptionkey@0"
I20260927 10:44:18.247273 8921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "59cc9eb99c984b3399dde1c2e4a0220f"
format_stamp: "Formatted at 2026-09-27 10:44:18 on dist-test-slave-dzhw"
server_key: "5a161aa0eafc1767f2b0336923c3b670"
server_key_iv: "e0b409951d1b80aaa5488ea2fda6cc03"
server_key_version: "encryptionkey@0"
I20260927 10:44:18.251946 8921 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260927 10:44:18.254783 8936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:18.256088 8921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:44:18.256212 8921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "59cc9eb99c984b3399dde1c2e4a0220f"
format_stamp: "Formatted at 2026-09-27 10:44:18 on dist-test-slave-dzhw"
server_key: "5a161aa0eafc1767f2b0336923c3b670"
server_key_iv: "e0b409951d1b80aaa5488ea2fda6cc03"
server_key_version: "encryptionkey@0"
I20260927 10:44:18.256316 8921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:18.269304 8921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:18.272794 8921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:18.272954 8921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:18.273486 8921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:18.274359 8921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:18.274408 8921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:18.274478 8921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:18.274494 8921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:18.283660 8921 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45185
I20260927 10:44:18.283684 9049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45185 every 8 connection(s)
I20260927 10:44:18.284590 8921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:44:18.293723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 8921
I20260927 10:44:18.293882 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:44:18.294289 392 external_mini_cluster.cc:1428] Setting key 703c308ac0d63d4dd89a194309e99c5a
Sep 27 10:44:18 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:44:18.298983 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:18.286378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59795 (local address 127.0.98.62:36955)
0927 10:44:18.286549 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:18.286553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:18.287401 (+ 848us) server_negotiation.cc:407] Connection header received
0927 10:44:18.288881 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:18.288884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:18.288943 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:18.289051 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:18.290199 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.290788 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.291977 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.292248 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.292363 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:18.295146 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:18.295152 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:18.295158 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:18.295188 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:18.296859 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:18.297392 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:18.297397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:18.297399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:18.297482 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:18.297827 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:18.297832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:18.297834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:18.298016 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:18.298135 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:18.298705 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 10:44:18.298825 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 10:44:18.299821 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:18.286811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.3:59795)
0927 10:44:18.287252 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:18.287294 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:18.288657 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:18.289228 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:18.289234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:18.289587 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:18.290052 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:18.290064 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.290910 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.290913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:18.291814 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:18.291828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.292408 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.292411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:18.292583 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:18.293182 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:18.293198 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:18.294962 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:18.296983 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:18.296989 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:18.297005 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:18.297286 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:18.297574 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:18.297578 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:18.297580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:18.297731 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:18.298123 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:18.298130 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:18.298381 (+ 251us) client_negotiation.cc:769] Sending connection context
0927 10:44:18.298666 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 10:44:18.298999 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":116,"threads_started":1}
I20260927 10:44:18.300932 9050 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36955
I20260927 10:44:18.301172 9050 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:18.301654 9050 heartbeater.cc:511] Master 127.0.98.62:36955 requested a full tablet report, sending...
I20260927 10:44:18.302716 8590 ts_manager.cc:194] Registered new tserver with Master: 59cc9eb99c984b3399dde1c2e4a0220f (127.0.98.3:45185)
I20260927 10:44:18.303301 8590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59795
I20260927 10:44:18.308265 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 10:44:18.383533 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 8653
I20260927 10:44:18.389286 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 8787
I20260927 10:44:18.394642 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 8921
I20260927 10:44:18.400110 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 8559
I20260927 10:44:18.406409 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36955
--webserver_interface=127.0.98.62
--webserver_port=39103
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:18.515579 9059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:18.515872 9059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:18.515913 9059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:18.519985 9059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:44:18.520054 9059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:18.520074 9059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:18.520092 9059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:44:18.520116 9059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:44:18.525424 9059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:18.526501 9059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:18.527519 9059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:18.536044 9067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:18.536000 9065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:18.536257 9059 server_base.cc:1034] running on GCE node
W20260927 10:44:18.536023 9064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:18.536887 9059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:18.537988 9059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:18.539155 9059 hybrid_clock.cc:648] HybridClock initialized: now 1790505858539139 us; error 29 us; skew 500 ppm
Sep 27 10:44:18 dist-test-slave-dzhw krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:18.541986 9059 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:44:18.542939 9059 webserver.cc:460] Webserver started at http://127.0.98.62:39103/ using document root <none> and password file <none>
I20260927 10:44:18.543505 9059 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:18.543576 9059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:18.547668 9059 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:44:18.550005 9074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:18.551368 9059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:44:18.551512 9059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "395d5988443d41fb9d078e20ad0a2366"
format_stamp: "Formatted at 2026-09-27 10:44:17 on dist-test-slave-dzhw"
server_key: "51f534e09ace45f8ce35581c1be1a520"
server_key_iv: "dcee0d00c9a0d9cfde524230ac77e19c"
server_key_version: "encryptionkey@0"
I20260927 10:44:18.552069 9059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:18.569878 9059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:18.572857 9059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:18.573071 9059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:18.579993 9059 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36955
I20260927 10:44:18.580014 9126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36955 every 8 connection(s)
I20260927 10:44:18.581032 9059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:44:18.582686 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 9059
I20260927 10:44:18.583186 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36753
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36955
--webserver_port=40815
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:44:18.585850 9127 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:44:18.587730 9127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366: Bootstrap starting.
I20260927 10:44:18.604929 9127 log.cc:824] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:18.611274 9127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:44:18.611662 9127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366: Bootstrap complete.
I20260927 10:44:18.615972 9127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395d5988443d41fb9d078e20ad0a2366" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36955 } }
I20260927 10:44:18.616358 9127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395d5988443d41fb9d078e20ad0a2366, State: Initialized, Role: FOLLOWER
I20260927 10:44:18.616889 9127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395d5988443d41fb9d078e20ad0a2366" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36955 } }
I20260927 10:44:18.617002 9127 raft_consensus.cc:388] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:44:18.617090 9127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:44:18.617185 9127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:44:18.619205 9127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395d5988443d41fb9d078e20ad0a2366" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36955 } }
I20260927 10:44:18.619531 9127 leader_election.cc:302] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 395d5988443d41fb9d078e20ad0a2366; no voters:
I20260927 10:44:18.619885 9127 leader_election.cc:288] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 10:44:18.620011 9131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 10:44:18.620222 9131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [term 2 LEADER]: Becoming Leader. State: Replica: 395d5988443d41fb9d078e20ad0a2366, State: Running, Role: LEADER
I20260927 10:44:18.620502 9131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395d5988443d41fb9d078e20ad0a2366" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36955 } }
I20260927 10:44:18.621081 9127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:44:18.621748 9132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "395d5988443d41fb9d078e20ad0a2366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395d5988443d41fb9d078e20ad0a2366" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36955 } } }
I20260927 10:44:18.621876 9132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:18.622610 9133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 395d5988443d41fb9d078e20ad0a2366. Latest consensus state: current_term: 2 leader_uuid: "395d5988443d41fb9d078e20ad0a2366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395d5988443d41fb9d078e20ad0a2366" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36955 } } }
I20260927 10:44:18.622699 9133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366 [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:18.622865 9137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:44:18.625893 9137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:44:18.626528 9137 catalog_manager.cc:1259] Loaded cluster ID: 4bc9e34dd59a4dc88a38b18fc067088d
I20260927 10:44:18.626585 9137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:44:18.628870 9137 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:44:18.629522 9137 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 395d5988443d41fb9d078e20ad0a2366: Loaded TSK: 0
I20260927 10:44:18.630769 9137 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:44:18.698289 9129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:18.698678 9129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:18.698774 9129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:18.702852 9129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:18.702937 9129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:18.703027 9129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:44:18.708300 9129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36753
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40815
--tserver_master_addrs=127.0.98.62:36955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:18.709373 9129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:18.710352 9129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:18.720070 9155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:18.720036 9157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:18.720053 9154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:18.720711 9129 server_base.cc:1034] running on GCE node
I20260927 10:44:18.721163 9129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:18.721905 9129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:18.723079 9129 hybrid_clock.cc:648] HybridClock initialized: now 1790505858723028 us; error 69 us; skew 500 ppm
Sep 27 10:44:18 dist-test-slave-dzhw krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505858, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:18.725854 9129 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 10:44:18.726864 9129 webserver.cc:460] Webserver started at http://127.0.98.1:40815/ using document root <none> and password file <none>
I20260927 10:44:18.727455 9129 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:18.727528 9129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:18.731479 9129 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:44:18.733628 9164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:18.734958 9129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:44:18.735059 9129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "5a4d74ef143d40ddbdab2db64ff5a9ad"
format_stamp: "Formatted at 2026-09-27 10:44:17 on dist-test-slave-dzhw"
server_key: "955159a1a2d1bc9957c9460185e26322"
server_key_iv: "b72e9b58c135e4eba8c221ba7fa2f60b"
server_key_version: "encryptionkey@0"
I20260927 10:44:18.735601 9129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:18.766000 9129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:18.768862 9129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:18.769066 9129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:18.769670 9129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:18.770514 9129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:18.770560 9129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:18.770629 9129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:18.770659 9129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:18.779487 9129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36753
I20260927 10:44:18.779512 9277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36753 every 8 connection(s)
I20260927 10:44:18.780485 9129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:44:18.790326 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 9129
Sep 27 10:44:18 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:18.790889 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34865
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36955
--webserver_port=34655
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:44:18 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:18.793366 9281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:18.782710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.1:41693)
0927 10:44:18.783181 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:18.783220 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:18.784402 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:18.785419 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:18.785425 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:18.785776 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:18.786239 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:18.786253 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.787499 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.787502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:18.788374 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:18.788383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.789101 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.789104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:18.789246 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:18.789825 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:18.789842 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:18.792587 (+ 2745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":135,"threads_started":1}
W20260927 10:44:18.793885 9278 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36955 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36955: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260927 10:44:18.794296 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:18.782559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41693 (local address 127.0.98.62:36955)
0927 10:44:18.782999 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:18.783018 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:18.783330 (+ 312us) server_negotiation.cc:407] Connection header received
0927 10:44:18.784752 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:18.784762 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:18.785041 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:18.785300 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:18.786390 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.787362 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.788478 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.788925 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.789042 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:18.792881 (+ 3839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41693: BlockingRecv error: recv got EOF from 127.0.98.1:41693 (error 108)
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":93,"threads_started":1}
Sep 27 10:44:18 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:18 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:18.800344 9281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:18.794354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.1:34149)
0927 10:44:18.794563 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:18.794579 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:18.794658 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:18.794965 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:18.794968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:18.795103 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:18.795270 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:18.795277 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.796455 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.796458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:18.796991 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:18.796997 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.797662 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.797665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:18.797732 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:18.798230 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:18.798241 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:18.800157 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129}
W20260927 10:44:18.800372 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:18.794470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34149 (local address 127.0.98.62:36955)
0927 10:44:18.794626 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:18.794630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:18.794647 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:44:18.794778 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:18.794784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:18.794844 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:18.794930 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:18.795400 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.796339 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.797106 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.797530 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.797663 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:18.800278 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34149: BlockingRecv error: recv got EOF from 127.0.98.1:34149 (error 108)
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":123,"threads_started":1}
Sep 27 10:44:18 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:18 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:18.806653 9281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:18.801052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.1:50483)
0927 10:44:18.801302 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:18.801320 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:18.801424 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:18.801668 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:18.801673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:18.801857 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:18.801978 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:18.801987 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.802748 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.802752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:18.803619 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:18.803640 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.804125 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.804129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:18.804210 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:18.804689 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:18.804703 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:18.806514 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158}
W20260927 10:44:18.806746 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:18.801145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50483 (local address 127.0.98.62:36955)
0927 10:44:18.801269 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:18.801272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:18.801386 (+ 114us) server_negotiation.cc:407] Connection header received
0927 10:44:18.801515 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:18.801517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:18.801553 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:18.801616 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:18.802101 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.802650 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.803765 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.803985 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.804105 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:18.806630 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50483: BlockingRecv error: recv got EOF from 127.0.98.1:50483 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 10:44:18.807020 9278 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:44:18.905143 9282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:18.905472 9282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:18.905553 9282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:18.909708 9282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:18.909806 9282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:18.909899 9282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:44:18.915357 9282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34655
--tserver_master_addrs=127.0.98.62:36955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:18.916512 9282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:18.917583 9282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:18.927240 9291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:18.927302 9289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:18.927240 9288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:18.927580 9282 server_base.cc:1034] running on GCE node
I20260927 10:44:18.927973 9282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:18.928642 9282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:18.929834 9282 hybrid_clock.cc:648] HybridClock initialized: now 1790505858929819 us; error 31 us; skew 500 ppm
Sep 27 10:44:18 dist-test-slave-dzhw krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505858, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:18.932539 9282 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 10:44:18.933488 9282 webserver.cc:460] Webserver started at http://127.0.98.2:34655/ using document root <none> and password file <none>
I20260927 10:44:18.934006 9282 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:18.934101 9282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:18.937793 9282 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:44:18.940019 9298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:18.941453 9282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 10:44:18.941558 9282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "0e9d0937a32f4426a4e935bebf0054e0"
format_stamp: "Formatted at 2026-09-27 10:44:17 on dist-test-slave-dzhw"
server_key: "11c55b12cb38d43edc904a7779e279dc"
server_key_iv: "96f508cdef0472d9a055239703e29c29"
server_key_version: "encryptionkey@0"
I20260927 10:44:18.942073 9282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:18.957301 9282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:18.960148 9282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:18.960304 9282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:18.960844 9282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:18.961621 9282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:18.961666 9282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:18.961705 9282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:18.961727 9282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:18.970605 9282 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34865
I20260927 10:44:18.970628 9411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34865 every 8 connection(s)
I20260927 10:44:18.971592 9282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:44:18.978313 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 9282
I20260927 10:44:18.978818 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45185
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36955
--webserver_port=34053
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:44:18 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:18 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 10:44:18.982982 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:18.973305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48609 (local address 127.0.98.62:36955)
0927 10:44:18.973469 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:18.973474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:18.974274 (+ 800us) server_negotiation.cc:407] Connection header received
0927 10:44:18.975591 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:18.975594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:18.975644 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:18.975719 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:18.976872 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.977402 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.978528 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.978897 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.979023 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:18.982865 (+ 3842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48609: BlockingRecv error: recv got EOF from 127.0.98.2:48609 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 10:44:18.983963 9414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:18.973692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.2:48609)
0927 10:44:18.974123 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:18.974163 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:18.975405 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:18.975886 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:18.975892 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:18.976246 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:18.976714 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:18.976734 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.977511 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.977513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:18.978346 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:18.978354 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.979135 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.979138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:18.979276 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:18.979894 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:18.979922 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:18.982749 (+ 2827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":115,"threads_started":1}
W20260927 10:44:18.984447 9412 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36955 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36955: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 10:44:18 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:18 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:18.989773 9414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:18.984902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.2:44485)
0927 10:44:18.984996 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:18.985011 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:18.985101 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:18.985329 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:18.985332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:18.985499 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:18.985582 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:18.985587 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.986296 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.986303 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:18.986892 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:18.986900 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.987353 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.987356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:18.987444 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:18.987959 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:18.987971 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:18.989629 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 10:44:18.989856 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:18.984997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44485 (local address 127.0.98.62:36955)
0927 10:44:18.985123 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:18.985129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:18.985145 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:44:18.985186 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:18.985189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:18.985231 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:18.985315 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:18.985688 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.986190 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.987021 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.987237 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.987344 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:18.989742 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44485: BlockingRecv error: recv got EOF from 127.0.98.2:44485 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 10:44:18 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:18 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:18.995404 9414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:18.990409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.2:53499)
0927 10:44:18.990531 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:18.990546 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:18.990617 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:18.990868 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:18.990870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:18.990986 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:18.991064 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:18.991068 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.992086 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.992089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:18.992586 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:18.992594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.993142 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.993144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:18.993200 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:18.993701 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:18.993712 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:18.995316 (+ 1604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 10:44:18.995651 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:18.990504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53499 (local address 127.0.98.62:36955)
0927 10:44:18.990608 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:18.990612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:18.990626 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:44:18.990714 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:18.990723 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:18.990763 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:18.990848 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:18.991159 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.991945 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.992696 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:18.993039 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:18.993135 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:18.995547 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53499: BlockingRecv error: recv got EOF from 127.0.98.2:53499 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 10:44:18.995728 9412 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:44:19.090633 9415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:19.090924 9415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:19.091001 9415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:19.095477 9415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:19.095584 9415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:19.095681 9415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:44:19.101058 9415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45185
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34053
--tserver_master_addrs=127.0.98.62:36955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:19.102149 9415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:19.103185 9415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:19.112654 9421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:19.112720 9420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:19.112674 9423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:19.112773 9415 server_base.cc:1034] running on GCE node
I20260927 10:44:19.113300 9415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:19.114022 9415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:19.115243 9415 hybrid_clock.cc:648] HybridClock initialized: now 1790505859115229 us; error 31 us; skew 500 ppm
Sep 27 10:44:19 dist-test-slave-dzhw krb5kdc[8546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505859, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:19.118116 9415 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 10:44:19.119077 9415 webserver.cc:460] Webserver started at http://127.0.98.3:34053/ using document root <none> and password file <none>
I20260927 10:44:19.119663 9415 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:19.119771 9415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:19.124085 9415 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:44:19.126175 9430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:19.127517 9415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:44:19.127671 9415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "59cc9eb99c984b3399dde1c2e4a0220f"
format_stamp: "Formatted at 2026-09-27 10:44:18 on dist-test-slave-dzhw"
server_key: "5a161aa0eafc1767f2b0336923c3b670"
server_key_iv: "e0b409951d1b80aaa5488ea2fda6cc03"
server_key_version: "encryptionkey@0"
I20260927 10:44:19.128221 9415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:19.144281 9415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:19.147686 9415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:19.147902 9415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:19.148490 9415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:19.149355 9415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:19.149402 9415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:19.149469 9415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:19.149509 9415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:19.158115 9415 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45185
I20260927 10:44:19.158162 9543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45185 every 8 connection(s)
I20260927 10:44:19.159127 9415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:44:19.165642 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 9415
Sep 27 10:44:19 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:19 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 10:44:19.170480 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:19.160932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47257 (local address 127.0.98.62:36955)
0927 10:44:19.161086 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:19.161090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:19.161964 (+ 874us) server_negotiation.cc:407] Connection header received
0927 10:44:19.163327 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:19.163330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:19.163377 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:19.163455 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:19.164586 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:19.165138 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:19.166313 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:19.166595 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:19.166710 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:19.170326 (+ 3616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47257: BlockingRecv error: recv got EOF from 127.0.98.3:47257 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:44:19.171119 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:19.161372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.3:47257)
0927 10:44:19.161810 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:19.161851 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:19.163112 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:19.163614 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:19.163620 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:19.163972 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:19.164450 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:19.164463 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:19.165251 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:19.165253 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:19.166179 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:19.166188 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:19.166761 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:19.166768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:19.166982 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:19.167612 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:19.167636 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:19.170247 (+ 2611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":118,"threads_started":1}
W20260927 10:44:19.171689 9544 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36955 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36955: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260927 10:44:19.175668 9283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:19.166722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50454 (local address 127.0.98.62:36955)
0927 10:44:19.166847 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:19.166857 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:19.166868 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:44:19.166957 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:19.166959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:19.167003 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:19.167074 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:19.167962 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:19.168620 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:19.169574 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:19.169940 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:19.170046 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:19.171632 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:19.171638 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:19.171654 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:19.171683 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:19.173593 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:19.173992 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:19.173998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:19.174004 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:19.174141 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:19.174387 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:19.174390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:19.174392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:19.174824 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:19.175018 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:19.175273 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 10:44:19.175513 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 10:44:19 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:19 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:19.177229 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:19.172140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.3:41495)
0927 10:44:19.172271 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:19.172284 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:19.172359 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:19.172604 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:19.172609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:19.172786 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:19.172911 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:19.172920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:19.173599 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:19.173603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:19.174391 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:19.174401 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:19.174824 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:19.174828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:19.174929 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:19.175358 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:19.175370 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:19.177077 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 10:44:19.177274 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:19.172212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41495 (local address 127.0.98.62:36955)
0927 10:44:19.172327 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:19.172332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:19.172350 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:44:19.172449 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:19.172451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:19.172484 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:19.172554 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:19.173026 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:19.173482 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:19.174516 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:19.174716 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:19.174790 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:19.177181 (+ 2391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41495: BlockingRecv error: recv got EOF from 127.0.98.3:41495 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 10:44:19 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:19 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:19.182550 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:19.178156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.3:34795)
0927 10:44:19.178258 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:19.178272 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:19.178347 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:19.178692 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:19.178694 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:19.178815 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:19.178877 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:19.178881 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:19.179546 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:19.179548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:19.180039 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:19.180045 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:19.180411 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:19.180414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:19.180465 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:19.180835 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:19.180845 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:19.182452 (+ 1607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 10:44:19.182682 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:19.178225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34795 (local address 127.0.98.62:36955)
0927 10:44:19.178348 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:19.178353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:19.178367 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:44:19.178450 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:19.178453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:19.178496 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:19.178581 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:19.178972 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:19.179455 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:19.180130 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:19.180324 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:19.180396 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:19.182576 (+ 2180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34795: BlockingRecv error: recv got EOF from 127.0.98.3:34795 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 10:44:19.182786 9544 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 10:44:19 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:19 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:19.814023 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:19.807764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.1:55141)
0927 10:44:19.808099 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:19.808119 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:19.808238 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:19.808554 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:19.808557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:19.808738 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:19.808907 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:19.808916 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:19.809909 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:19.809913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:19.810873 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:19.810883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:19.811298 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:19.811301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:19.811375 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:19.811921 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:19.811935 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:19.813875 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":85,"threads_started":1}
W20260927 10:44:19.814278 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:19.807863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55141 (local address 127.0.98.62:36955)
0927 10:44:19.808202 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:19.808205 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:19.808222 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:44:19.808350 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:19.808353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:19.808395 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:19.808476 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:19.809027 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:19.809784 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:19.810977 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:19.811191 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:19.811274 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:19.814119 (+ 2845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55141: BlockingRecv error: recv got EOF from 127.0.98.1:55141 (error 108)
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":138,"threads_started":1}
Sep 27 10:44:19 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:19 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:20.002323 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:19.996486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.2:42115)
0927 10:44:19.996751 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:19.996768 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:19.996872 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:19.997215 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:19.997218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:19.997438 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:19.997602 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:19.997612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:19.998617 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:19.998622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:19.999212 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:19.999221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:19.999625 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:19.999628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:19.999697 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:20.000255 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:20.000269 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:20.002137 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":100,"threads_started":1}
W20260927 10:44:20.002396 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:19.996661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42115 (local address 127.0.98.62:36955)
0927 10:44:19.996813 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:19.996816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:19.996832 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:44:19.996996 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:19.996999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:19.997053 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:19.997157 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:19.997737 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:19.998505 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:19.999323 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:19.999530 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:19.999606 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:20.002280 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42115: BlockingRecv error: recv got EOF from 127.0.98.2:42115 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 10:44:20 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:20 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:20.190289 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:20.183503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.3:48537)
0927 10:44:20.183811 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:20.183830 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:20.183950 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:20.184287 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:20.184291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:20.184529 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:20.184702 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:20.184710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:20.185698 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:20.185718 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:20.186488 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:20.186501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:20.187120 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:20.187124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:20.187243 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:20.188007 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:20.188023 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:20.190153 (+ 2130us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":97,"threads_started":1}
W20260927 10:44:20.190464 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:20.183565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48537 (local address 127.0.98.62:36955)
0927 10:44:20.183792 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:20.183823 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:20.183919 (+ 96us) server_negotiation.cc:407] Connection header received
0927 10:44:20.184058 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:20.184061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:20.184119 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:20.184243 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:20.184845 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:20.185571 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:20.186639 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:20.186995 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:20.187111 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:20.190303 (+ 3192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48537: BlockingRecv error: recv got EOF from 127.0.98.3:48537 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 27 10:44:20 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:20 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:20.820585 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:20.814958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.1:33809)
0927 10:44:20.815312 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:20.815327 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:20.815426 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:20.815691 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:20.815694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:20.815894 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:20.815991 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:20.815996 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:20.816797 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:20.816800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:20.817506 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:20.817518 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:20.817935 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:20.817937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:20.818040 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:20.818644 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:20.818656 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:20.820450 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":93,"threads_started":1}
W20260927 10:44:20.820811 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:20.815119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33809 (local address 127.0.98.62:36955)
0927 10:44:20.815373 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:20.815376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:20.815392 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:44:20.815507 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:20.815509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:20.815550 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:20.815629 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:20.816118 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:20.816686 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:20.817619 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:20.817836 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:20.817919 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:20.820680 (+ 2761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33809: BlockingRecv error: recv got EOF from 127.0.98.1:33809 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":78,"threads_started":1}
Sep 27 10:44:21 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:21 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:21.009220 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:21.003375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.2:51103)
0927 10:44:21.003760 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:21.003775 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:21.003928 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:21.004208 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:21.004211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:21.004420 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:21.004509 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:21.004515 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:21.005274 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:21.005280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:21.005836 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:21.005844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:21.006300 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:21.006303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:21.006384 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:21.007031 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:21.007043 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:21.009027 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":93,"threads_started":1}
W20260927 10:44:21.009279 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:21.003533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51103 (local address 127.0.98.62:36955)
0927 10:44:21.003726 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:21.003731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:21.003865 (+ 134us) server_negotiation.cc:407] Connection header received
0927 10:44:21.004040 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:21.004043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:21.004093 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:21.004187 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:21.004638 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:21.005162 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:21.005958 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:21.006183 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:21.006295 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:21.009170 (+ 2875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51103: BlockingRecv error: recv got EOF from 127.0.98.2:51103 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 27 10:44:21 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:21 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:21.201265 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:21.191100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.3:57867)
0927 10:44:21.191433 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:21.191449 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:21.191596 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:21.191861 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:21.191863 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:21.192061 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:21.192169 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:21.192175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:21.192931 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:21.192937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:21.193487 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:21.193494 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:21.193944 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:21.193947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:21.194027 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:21.199181 (+ 5154us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:21.199195 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:21.201050 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":116,"threads_started":1}
W20260927 10:44:21.201287 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:21.191264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57867 (local address 127.0.98.62:36955)
0927 10:44:21.191433 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:21.191438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:21.191536 (+ 98us) server_negotiation.cc:407] Connection header received
0927 10:44:21.191696 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:21.191700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:21.191754 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:21.191841 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:21.192286 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:21.192823 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:21.193607 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:21.193834 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:21.193942 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:21.201190 (+ 7248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57867: BlockingRecv error: recv got EOF from 127.0.98.3:57867 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 27 10:44:21 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:21 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:21.827235 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:21.821432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.1:56099)
0927 10:44:21.821820 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:21.821834 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:21.821938 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:21.822300 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:21.822305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:21.822548 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:21.822657 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:21.822663 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:21.823469 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:21.823471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:21.824046 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:21.824054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:21.824517 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:21.824520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:21.824609 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:21.825234 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:21.825246 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:21.827096 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":101,"threads_started":1}
W20260927 10:44:21.827370 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:21.821481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56099 (local address 127.0.98.62:36955)
0927 10:44:21.821761 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:21.821765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:21.821909 (+ 144us) server_negotiation.cc:407] Connection header received
0927 10:44:21.822030 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:21.822033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:21.822077 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:21.822185 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:21.822788 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:21.823376 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:21.824160 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:21.824411 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:21.824489 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:21.827235 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56099: BlockingRecv error: recv got EOF from 127.0.98.1:56099 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":83,"threads_started":1}
Sep 27 10:44:22 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:22 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:22.015903 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:22.010134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.2:59493)
0927 10:44:22.010453 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:22.010467 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:22.010566 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:22.010867 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:22.010873 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:22.011075 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:22.011240 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:22.011248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:22.012288 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:22.012291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:22.012862 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:22.012870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:22.013273 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:22.013276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:22.013337 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:22.013940 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:22.013952 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:22.015783 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":119,"threads_started":1}
W20260927 10:44:22.016139 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:22.010270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59493 (local address 127.0.98.62:36955)
0927 10:44:22.010453 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:22.010460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:22.010543 (+ 83us) server_negotiation.cc:407] Connection header received
0927 10:44:22.010660 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:22.010663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:22.010722 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:22.010811 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:22.011365 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:22.012186 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:22.012963 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:22.013166 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:22.013239 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:22.015993 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59493: BlockingRecv error: recv got EOF from 127.0.98.2:59493 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 10:44:22 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:22 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:22.207715 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:22.202076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.3:60379)
0927 10:44:22.202404 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:22.202471 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:22.202614 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:22.202892 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:22.202895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:22.203104 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:22.203193 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:22.203199 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:22.203953 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:22.203959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:22.204518 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:22.204527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:22.204986 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:22.204989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:22.205067 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:22.205632 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:22.205644 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:22.207509 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":104,"threads_started":1}
W20260927 10:44:22.207733 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:22.202190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60379 (local address 127.0.98.62:36955)
0927 10:44:22.202362 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:22.202367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:22.202550 (+ 183us) server_negotiation.cc:407] Connection header received
0927 10:44:22.202715 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:22.202721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:22.202775 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:22.202871 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:22.203313 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:22.203844 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:22.204642 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:22.204874 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:22.204985 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:22.207634 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60379: BlockingRecv error: recv got EOF from 127.0.98.3:60379 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 27 10:44:22 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:22 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:22.833890 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:22.828138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.1:38747)
0927 10:44:22.828460 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:22.828474 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:22.828611 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:22.828939 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:22.828942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:22.829166 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:22.829301 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:22.829307 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:22.830103 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:22.830106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:22.830742 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:22.830751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:22.831178 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:22.831180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:22.831241 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:22.831765 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:22.831777 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:22.833761 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":109,"threads_started":1}
W20260927 10:44:22.834091 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:22.828177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38747 (local address 127.0.98.62:36955)
0927 10:44:22.828484 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:22.828487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:22.828602 (+ 115us) server_negotiation.cc:407] Connection header received
0927 10:44:22.828684 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:22.828686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:22.828750 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:22.828850 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:22.829439 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:22.829971 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:22.830841 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:22.831059 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:22.831147 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:22.833911 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38747: BlockingRecv error: recv got EOF from 127.0.98.1:38747 (error 108)
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":81,"threads_started":1}
Sep 27 10:44:23 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:23 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:23.022574 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:23.016784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.2:39501)
0927 10:44:23.017111 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:23.017128 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:23.017292 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:23.017560 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:23.017562 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:23.017773 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:23.017861 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:23.017867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:23.018690 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:23.018695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:23.019249 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:23.019257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:23.019707 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:23.019710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:23.019785 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:23.020325 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:23.020335 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:23.022377 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":127,"threads_started":1}
W20260927 10:44:23.022639 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:23.016949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39501 (local address 127.0.98.62:36955)
0927 10:44:23.017114 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:23.017120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:23.017228 (+ 108us) server_negotiation.cc:407] Connection header received
0927 10:44:23.017396 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:23.017399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:23.017452 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:23.017541 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:23.017981 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:23.018587 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:23.019373 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:23.019596 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:23.019711 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:23.022532 (+ 2821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39501: BlockingRecv error: recv got EOF from 127.0.98.2:39501 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 10:44:23 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:23 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:23.213773 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:23.208591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.3:35155)
0927 10:44:23.208878 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:23.208891 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:23.208986 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:23.209248 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:23.209250 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:23.209427 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:23.209518 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:23.209522 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:23.210254 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:23.210257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:23.210914 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:23.210923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:23.211356 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:23.211359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:23.211416 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:23.211895 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:23.211906 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:23.213644 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":114,"threads_started":1}
W20260927 10:44:23.213915 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:23.208675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35155 (local address 127.0.98.62:36955)
0927 10:44:23.208804 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:23.208807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:23.208958 (+ 151us) server_negotiation.cc:407] Connection header received
0927 10:44:23.209092 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:23.209094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:23.209138 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:23.209208 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:23.209618 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:23.210132 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:23.211037 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:23.211255 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:23.211338 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:23.213793 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35155: BlockingRecv error: recv got EOF from 127.0.98.3:35155 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 10:44:23 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:23 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:23.840114 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:23.834718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.1:39643)
0927 10:44:23.835021 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:23.835039 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:23.835155 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:23.835472 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:23.835475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:23.835723 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:23.835831 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:23.835837 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:23.836585 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:23.836587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:23.837155 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:23.837163 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:23.837561 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:23.837563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:23.837623 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:23.838098 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:23.838109 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:23.839973 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":97,"threads_started":1}
W20260927 10:44:23.840379 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:23.834749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39643 (local address 127.0.98.62:36955)
0927 10:44:23.835035 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:23.835041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:23.835124 (+ 83us) server_negotiation.cc:407] Connection header received
0927 10:44:23.835267 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:23.835270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:23.835325 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:23.835426 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:23.835930 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:23.836487 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:23.837252 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:23.837460 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:23.837534 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:23.840249 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39643: BlockingRecv error: recv got EOF from 127.0.98.1:39643 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":124,"threads_started":1}
Sep 27 10:44:24 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:24 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:24.028707 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:24.023382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.2:59591)
0927 10:44:24.023695 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:24.023718 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:24.023828 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:24.024088 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:24.024091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:24.024322 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:24.024417 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:24.024422 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:24.025120 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:24.025122 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:24.025694 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:24.025701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:24.026141 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:24.026143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:24.026203 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:24.026710 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:24.026721 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:24.028570 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
W20260927 10:44:24.028846 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:24.023571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59591 (local address 127.0.98.62:36955)
0927 10:44:24.023725 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:24.023729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:24.023821 (+ 92us) server_negotiation.cc:407] Connection header received
0927 10:44:24.023928 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:24.023931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:24.023982 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:24.024072 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:24.024518 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:24.025029 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:24.025815 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:24.026041 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:24.026127 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:24.028717 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59591: BlockingRecv error: recv got EOF from 127.0.98.2:59591 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 10:44:24 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:24 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:24.220371 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:24.214712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.3:55415)
0927 10:44:24.215000 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:24.215045 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:24.215145 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:24.215372 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:24.215374 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:24.215558 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:24.215648 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:24.215653 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:24.216456 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:24.216459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:24.217073 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:24.217082 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:24.217549 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:24.217560 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:24.217635 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:24.218196 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:24.218207 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:24.220236 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":105,"threads_started":1}
W20260927 10:44:24.220592 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:24.214807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55415 (local address 127.0.98.62:36955)
0927 10:44:24.214946 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:24.214953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:24.215144 (+ 191us) server_negotiation.cc:407] Connection header received
0927 10:44:24.215211 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:24.215214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:24.215259 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:24.215333 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:24.215777 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:24.216332 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:24.217189 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:24.217415 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:24.217506 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:24.220400 (+ 2894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55415: BlockingRecv error: recv got EOF from 127.0.98.3:55415 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 10:44:24 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:24 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:24.846683 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:24.841005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.1:47551)
0927 10:44:24.841315 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:24.841328 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:24.841457 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:24.841795 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:24.841797 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:24.841970 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:24.842061 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:24.842066 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:24.842977 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:24.842979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:24.843542 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:24.843550 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:24.843988 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:24.843991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:24.844066 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:24.844650 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:24.844662 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:24.846563 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":109,"threads_started":1}
W20260927 10:44:24.847014 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:24.841083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47551 (local address 127.0.98.62:36955)
0927 10:44:24.841335 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:24.841339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:24.841398 (+ 59us) server_negotiation.cc:407] Connection header received
0927 10:44:24.841608 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:24.841614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:24.841675 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:24.841765 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:24.842181 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:24.842877 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:24.843655 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:24.843866 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:24.843945 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:24.846838 (+ 2893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47551: BlockingRecv error: recv got EOF from 127.0.98.1:47551 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":94,"threads_started":1}
Sep 27 10:44:25 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:25 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:25.035102 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:25.029488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.2:48699)
0927 10:44:25.029829 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:25.029844 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:25.029949 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:25.030173 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:25.030175 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:25.030359 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:25.030529 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:25.030536 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:25.031350 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:25.031353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:25.031913 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:25.031922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:25.032408 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:25.032411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:25.032471 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:25.033024 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:25.033043 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:25.034925 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":110,"threads_started":1}
W20260927 10:44:25.035148 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:25.029537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48699 (local address 127.0.98.62:36955)
0927 10:44:25.029691 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:25.029696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:25.029925 (+ 229us) server_negotiation.cc:407] Connection header received
0927 10:44:25.030029 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:25.030031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:25.030073 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:25.030138 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:25.030637 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:25.031247 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:25.032062 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:25.032290 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:25.032380 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:25.035049 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48699: BlockingRecv error: recv got EOF from 127.0.98.2:48699 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 10:44:25 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:25 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:25.226851 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:25.221196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.3:41651)
0927 10:44:25.221518 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:25.221553 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:25.221723 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:25.221998 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:25.222001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:25.222213 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:25.222356 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:25.222362 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:25.223170 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:25.223176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:25.223742 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:25.223750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:25.224233 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:25.224237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:25.224318 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:25.224797 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:25.224808 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:25.226675 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":87,"threads_started":1}
W20260927 10:44:25.226908 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:25.221320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41651 (local address 127.0.98.62:36955)
0927 10:44:25.221489 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:25.221493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:25.221647 (+ 154us) server_negotiation.cc:407] Connection header received
0927 10:44:25.221832 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:25.221835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:25.221883 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:25.221981 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:25.222524 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:25.223052 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:25.223874 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:25.224120 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:25.224231 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:25.226799 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41651: BlockingRecv error: recv got EOF from 127.0.98.3:41651 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 10:44:25 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:25 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:25.853034 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:25.847499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.1:53501)
0927 10:44:25.847849 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:25.847864 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:25.847964 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:25.848243 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:25.848245 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:25.848426 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:25.848529 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:25.848534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:25.849344 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:25.849346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:25.849922 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:25.849931 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:25.850398 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:25.850401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:25.850510 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:25.851071 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:25.851083 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:25.852909 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":74,"threads_started":1}
W20260927 10:44:25.853207 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:25.847589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53501 (local address 127.0.98.62:36955)
0927 10:44:25.847880 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:25.847884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:25.847911 (+ 27us) server_negotiation.cc:407] Connection header received
0927 10:44:25.848055 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:25.848057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:25.848100 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:25.848178 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:25.848662 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:25.849214 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:25.850047 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:25.850288 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:25.850372 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:25.853062 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53501: BlockingRecv error: recv got EOF from 127.0.98.1:53501 (error 108)
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":125,"threads_started":1}
Sep 27 10:44:26 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:26 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:26.041260 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:26.035820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.2:51145)
0927 10:44:26.036129 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:26.036142 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:26.036241 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:26.036454 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:26.036456 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:26.036690 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:26.036804 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:26.036811 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:26.037510 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:26.037512 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:26.038066 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:26.038074 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:26.038537 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:26.038540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:26.038616 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:26.039174 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:26.039185 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:26.041138 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":103,"threads_started":1}
W20260927 10:44:26.041460 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:26.035895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51145 (local address 127.0.98.62:36955)
0927 10:44:26.036045 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:26.036049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:26.036211 (+ 162us) server_negotiation.cc:407] Connection header received
0927 10:44:26.036307 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:26.036309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:26.036350 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:26.036417 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:26.036905 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:26.037417 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:26.038175 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:26.038389 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:26.038536 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:26.041321 (+ 2785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51145: BlockingRecv error: recv got EOF from 127.0.98.2:51145 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 10:44:26 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:26 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:26.238523 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:26.227795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.3:44937)
0927 10:44:26.228184 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:26.228198 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:26.228296 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:26.228634 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:26.228637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:26.228812 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:26.228904 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:26.228909 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:26.229669 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:26.229672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:26.230243 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:26.230251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:26.230803 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:26.230806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:26.230866 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:26.236350 (+ 5484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:26.236362 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:26.238340 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":103,"threads_started":1}
W20260927 10:44:26.238708 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:26.227945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44937 (local address 127.0.98.62:36955)
0927 10:44:26.228085 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:26.228089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:26.228265 (+ 176us) server_negotiation.cc:407] Connection header received
0927 10:44:26.228407 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:26.228409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:26.228482 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:26.228577 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:26.229006 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:26.229538 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:26.230348 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:26.230697 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:26.230776 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:26.238549 (+ 7773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44937: BlockingRecv error: recv got EOF from 127.0.98.3:44937 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 10:44:26 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:26 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:26.859231 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:26.853861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.1:51867)
0927 10:44:26.854136 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:26.854149 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:26.854233 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:26.854574 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:26.854576 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:26.854772 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:26.854865 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:26.854871 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:26.855689 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:26.855692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:26.856288 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:26.856297 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:26.856762 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:26.856764 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:26.856822 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:26.857335 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:26.857346 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:26.859110 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":105,"threads_started":1}
W20260927 10:44:26.859392 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:26.854049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51867 (local address 127.0.98.62:36955)
0927 10:44:26.854267 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:26.854270 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:26.854283 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:44:26.854317 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:26.854319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:26.854359 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:26.854451 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:26.854991 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:26.855599 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:26.856453 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:26.856668 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:26.856747 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:26.859260 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51867: BlockingRecv error: recv got EOF from 127.0.98.1:51867 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":82,"threads_started":1}
Sep 27 10:44:27 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:27 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:27.047524 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:27.042115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.2:45889)
0927 10:44:27.042379 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:27.042393 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:27.042510 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:27.042733 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:27.042735 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:27.042909 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:27.043004 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:27.043008 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:27.043842 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:27.043844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:27.044403 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:27.044410 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:27.044901 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:27.044907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:27.044972 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:27.045521 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:27.045532 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:27.047410 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":98,"threads_started":1}
W20260927 10:44:27.047693 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:27.042137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45889 (local address 127.0.98.62:36955)
0927 10:44:27.042287 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:27.042291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:27.042481 (+ 190us) server_negotiation.cc:407] Connection header received
0927 10:44:27.042588 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:27.042590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:27.042635 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:27.042702 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:27.043099 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:27.043745 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:27.044531 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:27.044758 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:27.044865 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:27.047539 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45889: BlockingRecv error: recv got EOF from 127.0.98.2:45889 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 10:44:27 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:27 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:27.244640 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:27.239280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.3:54831)
0927 10:44:27.239582 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:27.239595 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:27.239728 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:27.240042 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:27.240045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:27.240231 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:27.240360 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:27.240365 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:27.241119 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:27.241121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:27.241672 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:27.241679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:27.242078 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:27.242080 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:27.242141 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:27.242770 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:27.242782 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:27.244512 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":118,"threads_started":1}
W20260927 10:44:27.244813 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:27.239353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54831 (local address 127.0.98.62:36955)
0927 10:44:27.239484 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:27.239487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:27.239707 (+ 220us) server_negotiation.cc:407] Connection header received
0927 10:44:27.239842 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:27.239847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:27.239911 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:27.239993 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:27.240488 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:27.241006 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:27.241771 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:27.241977 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:27.242066 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:27.244654 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54831: BlockingRecv error: recv got EOF from 127.0.98.3:54831 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 10:44:27 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:27 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:27.865549 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:27.860116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.1:45995)
0927 10:44:27.860408 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:27.860422 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:27.860514 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:27.860758 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:27.860760 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:27.860934 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:27.861024 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:27.861029 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:27.861753 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:27.861756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:27.862308 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:27.862315 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:27.862798 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:27.862801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:27.862861 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:27.863424 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:27.863436 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:27.865375 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":116,"threads_started":1}
W20260927 10:44:27.865622 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:27.860176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45995 (local address 127.0.98.62:36955)
0927 10:44:27.860439 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:27.860443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:27.860474 (+ 31us) server_negotiation.cc:407] Connection header received
0927 10:44:27.860601 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:27.860604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:27.860647 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:27.860725 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:27.861131 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:27.861662 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:27.862467 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:27.862700 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:27.862785 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:27.865501 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45995: BlockingRecv error: recv got EOF from 127.0.98.1:45995 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":68,"threads_started":1}
Sep 27 10:44:28 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:28 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:28.053810 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:28.048269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.2:42203)
0927 10:44:28.048581 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:28.048610 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:28.048758 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:28.049024 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:28.049027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:28.049239 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:28.049327 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:28.049333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:28.050069 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:28.050074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:28.050666 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:28.050674 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:28.051118 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:28.051121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:28.051194 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:28.051784 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:28.051795 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:28.053647 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":93,"threads_started":1}
W20260927 10:44:28.053907 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:28.048402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42203 (local address 127.0.98.62:36955)
0927 10:44:28.048583 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:28.048588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:28.048697 (+ 109us) server_negotiation.cc:407] Connection header received
0927 10:44:28.048860 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:28.048863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:28.048914 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:28.049005 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:28.049443 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:28.049961 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:28.050784 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:28.051006 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:28.051123 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:28.053773 (+ 2650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42203: BlockingRecv error: recv got EOF from 127.0.98.2:42203 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 27 10:44:28 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:28 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:28.251140 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:28.245463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.3:58459)
0927 10:44:28.245730 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:28.245748 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:28.245858 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:28.246181 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:28.246184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:28.246540 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:28.246634 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:28.246641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:28.247422 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:28.247427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:28.247983 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:28.247990 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:28.248448 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:28.248451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:28.248527 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:28.249084 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:28.249096 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:28.250968 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":95,"threads_started":1}
W20260927 10:44:28.251236 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:28.245677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58459 (local address 127.0.98.62:36955)
0927 10:44:28.245818 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:28.245822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:28.245837 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:44:28.246009 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:28.246014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:28.246071 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:28.246159 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:28.246755 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:28.247307 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:28.248119 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:28.248339 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:28.248444 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:28.251115 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58459: BlockingRecv error: recv got EOF from 127.0.98.3:58459 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 10:44:28 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:28 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:28.871722 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:28.866354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.1:52671)
0927 10:44:28.866674 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:28.866688 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:28.866771 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:28.867036 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:28.867038 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:28.867218 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:28.867309 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:28.867314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:28.868046 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:28.868048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:28.868630 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:28.868638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:28.869079 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:28.869081 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:28.869155 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:28.869660 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:28.869671 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:28.871547 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":150,"threads_started":1}
W20260927 10:44:28.871793 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:28.866485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52671 (local address 127.0.98.62:36955)
0927 10:44:28.866712 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:28.866715 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:28.866731 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:44:28.866855 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:28.866857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:28.866897 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:28.867006 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:28.867412 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:28.867951 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:28.868739 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:28.868955 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:28.869039 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:28.871669 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52671: BlockingRecv error: recv got EOF from 127.0.98.1:52671 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":88,"threads_started":1}
Sep 27 10:44:29 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:29 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:29.060743 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:29.054720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.2:36209)
0927 10:44:29.055061 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:29.055078 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:29.055227 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:29.055542 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:29.055546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:29.055769 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:29.055862 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:29.055867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:29.056740 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:29.056748 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:29.057365 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:29.057374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:29.057873 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:29.057876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:29.057962 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:29.058618 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:29.058631 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:29.060554 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":116,"threads_started":1}
W20260927 10:44:29.060806 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:29.054861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36209 (local address 127.0.98.62:36955)
0927 10:44:29.055043 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:29.055048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:29.055189 (+ 141us) server_negotiation.cc:407] Connection header received
0927 10:44:29.055345 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:29.055349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:29.055403 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:29.055506 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:29.056045 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:29.056620 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:29.057504 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:29.057751 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:29.057870 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:29.060689 (+ 2819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36209: BlockingRecv error: recv got EOF from 127.0.98.2:36209 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 10:44:29 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:29 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:29.257841 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:29.252133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.3:44151)
0927 10:44:29.252478 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:29.252493 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:29.252596 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:29.252895 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:29.252898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:29.253095 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:29.253200 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:29.253206 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:29.254004 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:29.254010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:29.254783 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:29.254793 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:29.255245 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:29.255249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:29.255326 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:29.255874 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:29.255885 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:29.257722 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":93,"threads_started":1}
W20260927 10:44:29.258009 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:29.252355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44151 (local address 127.0.98.62:36955)
0927 10:44:29.252502 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:29.252505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:29.252625 (+ 120us) server_negotiation.cc:407] Connection header received
0927 10:44:29.252673 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:29.252675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:29.252723 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:29.252804 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:29.253340 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:29.253886 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:29.254896 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:29.255127 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:29.255239 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:29.257853 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44151: BlockingRecv error: recv got EOF from 127.0.98.3:44151 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 10:44:29 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:29 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:29.878532 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:29.872613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.1:52439)
0927 10:44:29.872893 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:29.872906 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:29.872995 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:29.873236 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:29.873239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:29.873415 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:29.873509 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:29.873514 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:29.874386 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:29.874388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:29.874957 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:29.874964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:29.875403 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:29.875406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:29.875461 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:29.876043 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:29.876054 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:29.878311 (+ 2257us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":107,"threads_started":1}
W20260927 10:44:29.878612 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:29.872760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52439 (local address 127.0.98.62:36955)
0927 10:44:29.873019 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:29.873022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:29.873036 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:44:29.873075 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:29.873077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:29.873116 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:29.873204 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:29.873656 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:29.874284 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:29.875067 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:29.875302 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:29.875390 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:29.878477 (+ 3087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52439: BlockingRecv error: recv got EOF from 127.0.98.1:52439 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":88,"threads_started":1}
Sep 27 10:44:30 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:30 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:30.066865 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:30.061591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.2:35273)
0927 10:44:30.061875 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:30.061889 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:30.061987 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:30.062223 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:30.062225 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:30.062404 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:30.062531 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:30.062536 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:30.063236 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:30.063238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:30.063805 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:30.063813 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:30.064288 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:30.064291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:30.064370 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:30.064921 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:30.064933 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:30.066752 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":109,"threads_started":1}
W20260927 10:44:30.067046 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:30.061678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35273 (local address 127.0.98.62:36955)
0927 10:44:30.061817 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:30.061821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:30.061961 (+ 140us) server_negotiation.cc:407] Connection header received
0927 10:44:30.062067 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:30.062070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:30.062112 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:30.062185 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:30.062646 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:30.063149 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:30.063929 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:30.064172 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:30.064284 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:30.066885 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35273: BlockingRecv error: recv got EOF from 127.0.98.2:35273 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 10:44:30 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:30 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:30.266068 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:30.258763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.3:50707)
0927 10:44:30.259139 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:30.259156 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:30.259288 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:30.259635 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:30.259637 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:30.259895 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:30.260032 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:30.260044 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:30.261264 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:30.261268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:30.262105 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:30.262115 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:30.262963 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:30.262967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:30.263054 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:30.263624 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:30.263643 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:30.265891 (+ 2248us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":78,"threads_started":1}
W20260927 10:44:30.266112 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:30.258873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50707 (local address 127.0.98.62:36955)
0927 10:44:30.259024 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:30.259032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:30.259242 (+ 210us) server_negotiation.cc:407] Connection header received
0927 10:44:30.259409 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:30.259414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:30.259479 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:30.259582 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:30.260215 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:30.261121 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:30.262244 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:30.262812 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:30.262958 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:30.266012 (+ 3054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50707: BlockingRecv error: recv got EOF from 127.0.98.3:50707 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 10:44:30 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:30 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:30.885435 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:30.879471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.1:60275)
0927 10:44:30.879979 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:30.880004 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:30.880206 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:30.880552 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:30.880556 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:30.880786 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:30.880892 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:30.880898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:30.881675 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:30.881678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:30.882274 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:30.882283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:30.882818 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:30.882820 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:30.882881 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:30.883382 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:30.883393 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:30.885316 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":86,"threads_started":1}
W20260927 10:44:30.885653 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:30.879550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60275 (local address 127.0.98.62:36955)
0927 10:44:30.879851 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:30.879854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:30.880169 (+ 315us) server_negotiation.cc:407] Connection header received
0927 10:44:30.880324 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:30.880327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:30.880391 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:30.880476 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:30.881003 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:30.881530 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:30.882464 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:30.882708 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:30.882792 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:30.885488 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60275: BlockingRecv error: recv got EOF from 127.0.98.1:60275 (error 108)
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":81,"threads_started":1}
Sep 27 10:44:31 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:31 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:31.072979 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:31.067744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.2:50959)
0927 10:44:31.067993 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:31.068007 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:31.068102 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:31.068435 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:31.068441 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:31.068661 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:31.068757 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:31.068762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:31.069467 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:31.069469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:31.070010 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:31.070016 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:31.070514 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:31.070517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:31.070581 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:31.071132 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:31.071144 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:31.072863 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":76,"threads_started":1}
W20260927 10:44:31.073166 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:31.067790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50959 (local address 127.0.98.62:36955)
0927 10:44:31.067937 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:31.067942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:31.068087 (+ 145us) server_negotiation.cc:407] Connection header received
0927 10:44:31.068222 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:31.068224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:31.068294 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:31.068377 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:31.068858 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:31.069375 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:31.070118 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:31.070345 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:31.070453 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:31.073007 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50959: BlockingRecv error: recv got EOF from 127.0.98.2:50959 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 10:44:31 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:31 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:31.277628 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:31.266947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.3:59165)
0927 10:44:31.267233 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:31.267247 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:31.267340 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:31.267781 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:31.267784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:31.268016 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:31.268170 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:31.268178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:31.269204 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:31.269207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:31.269750 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:31.269757 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:31.270211 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:31.270213 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:31.270273 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:31.275708 (+ 5435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:31.275722 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:31.277492 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":101,"threads_started":1}
W20260927 10:44:31.277837 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:31.267121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59165 (local address 127.0.98.62:36955)
0927 10:44:31.267263 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:31.267272 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:31.267292 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:44:31.267558 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:31.267561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:31.267619 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:31.267723 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:31.268311 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:31.269090 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:31.269866 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:31.270107 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:31.270192 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:31.277703 (+ 7511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59165: BlockingRecv error: recv got EOF from 127.0.98.3:59165 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 10:44:31 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:31 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:31.891836 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:31.886241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.1:33617)
0927 10:44:31.886594 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:31.886611 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:31.886750 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:31.887065 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:31.887068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:31.887272 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:31.887361 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:31.887366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:31.888134 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:31.888139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:31.888695 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:31.888703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:31.889181 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:31.889185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:31.889264 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:31.889754 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:31.889765 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:31.891719 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":80,"threads_started":1}
W20260927 10:44:31.891999 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:31.886293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33617 (local address 127.0.98.62:36955)
0927 10:44:31.886539 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:31.886544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:31.886693 (+ 149us) server_negotiation.cc:407] Connection header received
0927 10:44:31.886857 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:31.886860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:31.886951 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:31.887049 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:31.887474 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:31.888020 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:31.888818 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:31.889066 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:31.889176 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:31.891863 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33617: BlockingRecv error: recv got EOF from 127.0.98.1:33617 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":62,"threads_started":1}
Sep 27 10:44:32 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:32 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:32.079468 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:32.073864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.2:51079)
0927 10:44:32.074215 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:32.074234 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:32.074347 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:32.074653 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:32.074656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:32.074899 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:32.075017 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:32.075025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:32.075781 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:32.075786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:32.076346 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:32.076354 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:32.076813 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:32.076817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:32.076899 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:32.077426 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:32.077438 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:32.079291 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":113,"threads_started":1}
W20260927 10:44:32.079528 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:32.074046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51079 (local address 127.0.98.62:36955)
0927 10:44:32.074224 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:32.074228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:32.074320 (+ 92us) server_negotiation.cc:407] Connection header received
0927 10:44:32.074484 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:32.074487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:32.074542 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:32.074629 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:32.075142 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:32.075672 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:32.076468 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:32.076698 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:32.076810 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:32.079421 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51079: BlockingRecv error: recv got EOF from 127.0.98.2:51079 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 10:44:32 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:32 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:32.283905 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:32.278500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.3:47209)
0927 10:44:32.278794 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:32.278809 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:32.278910 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:32.279165 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:32.279167 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:32.279352 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:32.279444 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:32.279449 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:32.280151 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:32.280153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:32.280708 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:32.280715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:32.281204 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:32.281207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:32.281272 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:32.281948 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:32.281960 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:32.283775 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":82,"threads_started":1}
W20260927 10:44:32.284025 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:32.278670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47209 (local address 127.0.98.62:36955)
0927 10:44:32.278819 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:32.278822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:32.278862 (+ 40us) server_negotiation.cc:407] Connection header received
0927 10:44:32.279008 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:32.279010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:32.279056 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:32.279124 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:32.279540 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:32.280060 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:32.280837 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:32.281066 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:32.281164 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:32.283906 (+ 2742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47209: BlockingRecv error: recv got EOF from 127.0.98.3:47209 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 10:44:32 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:32 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:32.898108 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:32.892695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.1:52055)
0927 10:44:32.892967 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:32.892983 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:32.893072 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:32.893272 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:32.893273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:32.893444 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:32.893542 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:32.893547 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:32.894297 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:32.894299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:32.894935 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:32.894944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:32.895426 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:32.895430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:32.895495 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:32.896083 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:32.896096 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:32.897958 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":104,"threads_started":1}
W20260927 10:44:32.898258 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:32.892766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52055 (local address 127.0.98.62:36955)
0927 10:44:32.893057 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:32.893061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:32.893076 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:44:32.893140 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:32.893142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:32.893182 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:32.893262 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:32.893649 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:32.894190 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:32.895055 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:32.895293 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:32.895384 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:32.898110 (+ 2726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52055: BlockingRecv error: recv got EOF from 127.0.98.1:52055 (error 108)
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":100,"threads_started":1}
Sep 27 10:44:33 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:33 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:33.085526 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:33.080265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.2:58349)
0927 10:44:33.080615 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:33.080629 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:33.080724 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:33.080961 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:33.080963 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:33.081141 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:33.081235 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:33.081240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:33.081954 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:33.081956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:33.082525 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:33.082534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:33.082965 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:33.082968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:33.083064 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:33.083562 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:33.083580 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:33.085354 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":112,"threads_started":1}
W20260927 10:44:33.085578 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:33.080396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58349 (local address 127.0.98.62:36955)
0927 10:44:33.080538 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:33.080542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:33.080700 (+ 158us) server_negotiation.cc:407] Connection header received
0927 10:44:33.080804 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:33.080807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:33.080849 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:33.080924 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:33.081332 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:33.081864 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:33.082631 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:33.082834 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:33.082910 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:33.085474 (+ 2564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58349: BlockingRecv error: recv got EOF from 127.0.98.2:58349 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 10:44:33 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:33 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:33.290320 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:33.284742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.3:38129)
0927 10:44:33.285100 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:33.285113 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:33.285244 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:33.285532 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:33.285536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:33.285739 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:33.285826 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:33.285831 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:33.286670 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:33.286674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:33.287246 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:33.287253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:33.287716 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:33.287719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:33.287799 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:33.288317 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:33.288327 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:33.290184 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":82,"threads_started":1}
W20260927 10:44:33.290495 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:33.284846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38129 (local address 127.0.98.62:36955)
0927 10:44:33.284996 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:33.285002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:33.285189 (+ 187us) server_negotiation.cc:407] Connection header received
0927 10:44:33.285351 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:33.285355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:33.285409 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:33.285511 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:33.285935 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:33.286559 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:33.287376 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:33.287602 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:33.287714 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:33.290329 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38129: BlockingRecv error: recv got EOF from 127.0.98.3:38129 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 10:44:33 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:33 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:33.904260 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:33.899094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.1:35463)
0927 10:44:33.899410 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:33.899423 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:33.899511 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:33.899725 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:33.899727 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:33.899898 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:33.899991 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:33.899995 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:33.900726 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:33.900728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:33.901303 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:33.901310 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:33.901761 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:33.901763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:33.901819 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:33.902314 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:33.902325 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:33.904133 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":141,"threads_started":1}
W20260927 10:44:33.904400 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:33.899149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35463 (local address 127.0.98.62:36955)
0927 10:44:33.899400 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:33.899403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:33.899471 (+ 68us) server_negotiation.cc:407] Connection header received
0927 10:44:33.899577 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:33.899579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:33.899618 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:33.899692 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:33.900113 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:33.900634 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:33.901435 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:33.901663 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:33.901748 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:33.904270 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35463: BlockingRecv error: recv got EOF from 127.0.98.1:35463 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":63,"threads_started":1}
Sep 27 10:44:34 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:44:34 dist-test-slave-dzhw krb5kdc[8546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:44:34.091933 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:34.086587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36955 (local address 127.0.98.2:49599)
0927 10:44:34.086830 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:34.086887 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:34.086991 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:34.087242 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:34.087244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:34.087471 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:34.087591 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:34.087598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:34.088414 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:34.088418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:34.088979 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:34.088987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:34.089437 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:34.089441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:34.089519 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:34.090012 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:34.090024 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:34.091809 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36955: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
W20260927 10:44:34.092170 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:34.086638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49599 (local address 127.0.98.62:36955)
0927 10:44:34.086778 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:34.086784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:34.086974 (+ 190us) server_negotiation.cc:407] Connection header received
0927 10:44:34.087077 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:34.087079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:34.087135 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:34.087202 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:34.087705 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:34.088301 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:34.089100 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:34.089324 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:34.089438 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:34.091944 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49599: BlockingRecv error: recv got EOF from 127.0.98.2:49599 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:44:34.167529 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 9129
I20260927 10:44:34.173036 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 9282
I20260927 10:44:34.179221 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 9415
I20260927 10:44:34.186852 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 9059
2026-09-27T10:44:34Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17992 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T10:44:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:44:34Z Disabled control of system clock
I20260927 10:44:34.217361 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41307
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 10:44:34.328423 9610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:34.328706 9610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:34.328739 9610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:34.332857 9610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:44:34.332932 9610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:34.332948 9610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:34.332965 9610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:44:34.332981 9610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:44:34.337682 9610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41307
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:34.338742 9610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:34.340056 9610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:44:34.349123 9610 server_base.cc:1034] running on GCE node
W20260927 10:44:34.349077 9616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:34.349093 9618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:34.349081 9615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:34.350013 9610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:34.351207 9610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:34.352370 9610 hybrid_clock.cc:648] HybridClock initialized: now 1790505874352358 us; error 28 us; skew 500 ppm
I20260927 10:44:34.354221 9610 webserver.cc:460] Webserver started at http://127.0.98.62:37239/ using document root <none> and password file <none>
I20260927 10:44:34.354858 9610 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:34.354921 9610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:34.355163 9610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:34.357236 9610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "11e1df42d1b04412b2e183a6214fc2aa"
format_stamp: "Formatted at 2026-09-27 10:44:34 on dist-test-slave-dzhw"
server_key: "a85f352d2b3fec4a49d40f5a53cb6666"
server_key_iv: "42183c38658252c3749b83f2f24c92a9"
server_key_version: "encryptionkey@0"
I20260927 10:44:34.357791 9610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "11e1df42d1b04412b2e183a6214fc2aa"
format_stamp: "Formatted at 2026-09-27 10:44:34 on dist-test-slave-dzhw"
server_key: "a85f352d2b3fec4a49d40f5a53cb6666"
server_key_iv: "42183c38658252c3749b83f2f24c92a9"
server_key_version: "encryptionkey@0"
I20260927 10:44:34.361379 9610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:44:34.363785 9624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:34.365037 9610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:44:34.365159 9610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "11e1df42d1b04412b2e183a6214fc2aa"
format_stamp: "Formatted at 2026-09-27 10:44:34 on dist-test-slave-dzhw"
server_key: "a85f352d2b3fec4a49d40f5a53cb6666"
server_key_iv: "42183c38658252c3749b83f2f24c92a9"
server_key_version: "encryptionkey@0"
I20260927 10:44:34.365273 9610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:34.378674 9610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:34.379347 9610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:34.379550 9610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:34.386183 9610 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41307
I20260927 10:44:34.386193 9676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41307 every 8 connection(s)
I20260927 10:44:34.387184 9610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:44:34.390146 9677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:34.393447 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 9610
I20260927 10:44:34.393570 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:44:34.393868 392 external_mini_cluster.cc:1428] Setting key 82751f070115c66063fe257079e14c4c
I20260927 10:44:34.396198 9677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11e1df42d1b04412b2e183a6214fc2aa: Bootstrap starting.
I20260927 10:44:34.398391 9677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11e1df42d1b04412b2e183a6214fc2aa: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:34.399166 9677 log.cc:824] T 00000000000000000000000000000000 P 11e1df42d1b04412b2e183a6214fc2aa: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:34.401001 9677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11e1df42d1b04412b2e183a6214fc2aa: No bootstrap required, opened a new log
I20260927 10:44:34.403542 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:34.395346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48016 (local address 127.0.98.62:41307)
0927 10:44:34.396004 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:34.396020 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:34.396055 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:44:34.396742 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:34.396760 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:34.397158 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:34.397515 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:34.397947 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:34.399017 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:34.400111 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:34.400578 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:34.400744 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:34.401673 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:34.401678 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:34.401691 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:44:34.402082 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:34.402134 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:44:34.402154 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:34.402266 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:34.402391 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 10:44:34.402661 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":144,"threads_started":1}
I20260927 10:44:34.404297 9677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11e1df42d1b04412b2e183a6214fc2aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11e1df42d1b04412b2e183a6214fc2aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41307 } }
I20260927 10:44:34.404562 9677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11e1df42d1b04412b2e183a6214fc2aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:34.404613 9677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11e1df42d1b04412b2e183a6214fc2aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11e1df42d1b04412b2e183a6214fc2aa, State: Initialized, Role: FOLLOWER
I20260927 10:44:34.405076 9677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11e1df42d1b04412b2e183a6214fc2aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11e1df42d1b04412b2e183a6214fc2aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41307 } }
I20260927 10:44:34.405230 9677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11e1df42d1b04412b2e183a6214fc2aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:44:34.405316 9677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11e1df42d1b04412b2e183a6214fc2aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:44:34.405411 9677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11e1df42d1b04412b2e183a6214fc2aa [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:34.406528 9677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11e1df42d1b04412b2e183a6214fc2aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11e1df42d1b04412b2e183a6214fc2aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41307 } }
I20260927 10:44:34.406862 9677 leader_election.cc:302] T 00000000000000000000000000000000 P 11e1df42d1b04412b2e183a6214fc2aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11e1df42d1b04412b2e183a6214fc2aa; no voters:
I20260927 10:44:34.407214 9677 leader_election.cc:288] T 00000000000000000000000000000000 P 11e1df42d1b04412b2e183a6214fc2aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:44:34.407338 9682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11e1df42d1b04412b2e183a6214fc2aa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:34.407582 9682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11e1df42d1b04412b2e183a6214fc2aa [term 1 LEADER]: Becoming Leader. State: Replica: 11e1df42d1b04412b2e183a6214fc2aa, State: Running, Role: LEADER
I20260927 10:44:34.407909 9682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11e1df42d1b04412b2e183a6214fc2aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11e1df42d1b04412b2e183a6214fc2aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41307 } }
I20260927 10:44:34.408375 9677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11e1df42d1b04412b2e183a6214fc2aa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:44:34.409593 9684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11e1df42d1b04412b2e183a6214fc2aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11e1df42d1b04412b2e183a6214fc2aa. Latest consensus state: current_term: 1 leader_uuid: "11e1df42d1b04412b2e183a6214fc2aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11e1df42d1b04412b2e183a6214fc2aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41307 } } }
I20260927 10:44:34.409618 9683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11e1df42d1b04412b2e183a6214fc2aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11e1df42d1b04412b2e183a6214fc2aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11e1df42d1b04412b2e183a6214fc2aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41307 } } }
I20260927 10:44:34.409708 9684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11e1df42d1b04412b2e183a6214fc2aa [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:34.409718 9683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11e1df42d1b04412b2e183a6214fc2aa [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:34.410072 9691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:44:34.412763 9691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:44:34.418286 9691 catalog_manager.cc:1347] Generated new cluster ID: c63c029180674f61bbfe9eb5aca5399c
I20260927 10:44:34.418362 9691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:44:34.434690 9691 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:44:34.435951 9691 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:44:34.443399 9691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11e1df42d1b04412b2e183a6214fc2aa: Generated new TSK 0
I20260927 10:44:34.444203 9691 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:44:34.449645 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41307
--builtin_ntp_servers=127.0.98.20:43823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 10:44:34.561142 9701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:34.561448 9701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:34.561513 9701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:34.565606 9701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:34.565694 9701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:34.565783 9701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:44:34.570720 9701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:34.571827 9701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:34.573191 9701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:34.582710 9706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:34.582726 9709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:34.582706 9707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:34.583096 9701 server_base.cc:1034] running on GCE node
I20260927 10:44:34.583524 9701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:34.584237 9701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:34.585397 9701 hybrid_clock.cc:648] HybridClock initialized: now 1790505874585383 us; error 27 us; skew 500 ppm
I20260927 10:44:34.587375 9701 webserver.cc:460] Webserver started at http://127.0.98.1:35365/ using document root <none> and password file <none>
I20260927 10:44:34.587944 9701 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:34.588034 9701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:34.588272 9701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:34.590301 9701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "e9e38ea2d76d48a197934852f6f50aba"
format_stamp: "Formatted at 2026-09-27 10:44:34 on dist-test-slave-dzhw"
server_key: "6ffe500f461c4f73299b1d701d53963e"
server_key_iv: "862a0c7c6ed2a10ba6eb2b9587f6afc7"
server_key_version: "encryptionkey@0"
I20260927 10:44:34.590909 9701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e9e38ea2d76d48a197934852f6f50aba"
format_stamp: "Formatted at 2026-09-27 10:44:34 on dist-test-slave-dzhw"
server_key: "6ffe500f461c4f73299b1d701d53963e"
server_key_iv: "862a0c7c6ed2a10ba6eb2b9587f6afc7"
server_key_version: "encryptionkey@0"
I20260927 10:44:34.594523 9701 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:44:34.596998 9715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:34.598167 9701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:44:34.598300 9701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "e9e38ea2d76d48a197934852f6f50aba"
format_stamp: "Formatted at 2026-09-27 10:44:34 on dist-test-slave-dzhw"
server_key: "6ffe500f461c4f73299b1d701d53963e"
server_key_iv: "862a0c7c6ed2a10ba6eb2b9587f6afc7"
server_key_version: "encryptionkey@0"
I20260927 10:44:34.598418 9701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:34.623992 9701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:34.624702 9701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:34.624899 9701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:34.625474 9701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:34.626396 9701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:34.626482 9701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:34.626555 9701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:34.626608 9701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:34.635766 9701 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45759
I20260927 10:44:34.635792 9828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45759 every 8 connection(s)
I20260927 10:44:34.636651 9701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:44:34.645371 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:34.638520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47513 (local address 127.0.98.62:41307)
0927 10:44:34.638686 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:34.638691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:34.639715 (+ 1024us) server_negotiation.cc:407] Connection header received
0927 10:44:34.641012 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:34.641014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:34.641058 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:34.641134 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:34.641937 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:34.642539 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:34.643614 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:34.643825 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:34.643905 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:34.644991 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:34.644993 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:34.644995 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:44:34.645005 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:34.645019 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:44:34.645026 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:34.645077 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:34.645215 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 10:44:34.645261 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 10:44:34.646234 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:34.638940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41307 (local address 127.0.98.1:47513)
0927 10:44:34.639568 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:34.639607 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:34.640835 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:34.641275 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:34.641281 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:34.641360 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:34.641824 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:34.641835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:34.642646 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:34.642649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:34.643505 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:34.643513 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:34.643957 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:34.643959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:34.644109 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:34.644758 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:44:34.644760 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:34.644798 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:44:34.644859 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:34.645097 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:34.645103 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:34.645106 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 10:44:34.645199 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 10:44:34.645406 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":120,"threads_started":1}
I20260927 10:44:34.646410 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 9701
I20260927 10:44:34.646539 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:44:34.646885 392 external_mini_cluster.cc:1428] Setting key 45d47a256c36655903b1375a3779bc14
I20260927 10:44:34.647724 9829 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41307
I20260927 10:44:34.647949 9829 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:34.648469 9829 heartbeater.cc:511] Master 127.0.98.62:41307 requested a full tablet report, sending...
I20260927 10:44:34.648811 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41307
--builtin_ntp_servers=127.0.98.20:43823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 10:44:34.650015 9641 ts_manager.cc:194] Registered new tserver with Master: e9e38ea2d76d48a197934852f6f50aba (127.0.98.1:45759)
I20260927 10:44:34.651556 9641 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:47513
W20260927 10:44:34.758302 9832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:34.758641 9832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:34.758682 9832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:34.762745 9832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:34.762802 9832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:34.762856 9832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:44:34.767624 9832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:34.768739 9832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:34.770133 9832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:34.779610 9837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:34.779618 9838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:34.779714 9832 server_base.cc:1034] running on GCE node
W20260927 10:44:34.779647 9840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:34.780292 9832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:34.780915 9832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:34.782087 9832 hybrid_clock.cc:648] HybridClock initialized: now 1790505874782067 us; error 34 us; skew 500 ppm
I20260927 10:44:34.783977 9832 webserver.cc:460] Webserver started at http://127.0.98.2:41689/ using document root <none> and password file <none>
I20260927 10:44:34.784500 9832 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:34.784560 9832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:34.784718 9832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:34.786799 9832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "fd7885ded01e475fbd3430dcb1a6ad53"
format_stamp: "Formatted at 2026-09-27 10:44:34 on dist-test-slave-dzhw"
server_key: "33b9f2f67333fed12d0b4e64201b7920"
server_key_iv: "271234731070fcacdf2d0858197f9b4e"
server_key_version: "encryptionkey@0"
I20260927 10:44:34.787318 9832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fd7885ded01e475fbd3430dcb1a6ad53"
format_stamp: "Formatted at 2026-09-27 10:44:34 on dist-test-slave-dzhw"
server_key: "33b9f2f67333fed12d0b4e64201b7920"
server_key_iv: "271234731070fcacdf2d0858197f9b4e"
server_key_version: "encryptionkey@0"
I20260927 10:44:34.790973 9832 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:44:34.793455 9846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:34.794829 9832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:44:34.794937 9832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "fd7885ded01e475fbd3430dcb1a6ad53"
format_stamp: "Formatted at 2026-09-27 10:44:34 on dist-test-slave-dzhw"
server_key: "33b9f2f67333fed12d0b4e64201b7920"
server_key_iv: "271234731070fcacdf2d0858197f9b4e"
server_key_version: "encryptionkey@0"
I20260927 10:44:34.795111 9832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:34.812850 9832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:34.813592 9832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:34.813798 9832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:34.814399 9832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:34.815384 9832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:34.815457 9832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:34.815526 9832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:34.815578 9832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:34.824673 9832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35485
I20260927 10:44:34.824709 9959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35485 every 8 connection(s)
I20260927 10:44:34.825618 9832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:44:34.835233 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:34.827710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56643 (local address 127.0.98.62:41307)
0927 10:44:34.827863 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:34.827867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:34.829008 (+ 1141us) server_negotiation.cc:407] Connection header received
0927 10:44:34.830402 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:34.830405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:34.830530 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:34.830620 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:34.831534 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:34.832058 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:34.833229 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:34.833454 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:34.833554 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:34.834788 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:34.834791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:34.834793 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:44:34.834806 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:34.834825 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:44:34.834833 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:34.834893 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:34.835059 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 10:44:34.835115 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 10:44:34.835870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 9832
I20260927 10:44:34.835958 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:44:34.836253 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:34.828114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41307 (local address 127.0.98.2:56643)
0927 10:44:34.828828 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:34.828873 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:34.830185 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:34.830797 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:34.830804 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:34.830888 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:34.831394 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:34.831407 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:34.832207 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:34.832210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:34.833110 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:34.833120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:34.833623 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:34.833626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:34.833778 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:34.834524 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:44:34.834526 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:34.834568 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:44:34.834643 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:34.834930 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:34.834937 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:34.834941 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 10:44:34.835041 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 10:44:34.835269 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":122,"threads_started":1}
I20260927 10:44:34.836297 392 external_mini_cluster.cc:1428] Setting key 1993d8dc5919d4fb0721644e0a31530a
I20260927 10:44:34.837551 9960 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41307
I20260927 10:44:34.837908 9960 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:34.838563 9960 heartbeater.cc:511] Master 127.0.98.62:41307 requested a full tablet report, sending...
I20260927 10:44:34.838555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41307
--builtin_ntp_servers=127.0.98.20:43823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 10:44:34.839687 9640 ts_manager.cc:194] Registered new tserver with Master: fd7885ded01e475fbd3430dcb1a6ad53 (127.0.98.2:35485)
I20260927 10:44:34.840277 9640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:56643
W20260927 10:44:34.952092 9963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:34.952355 9963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:34.952394 9963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:34.956532 9963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:34.956597 9963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:34.956683 9963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:44:34.961511 9963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:34.962641 9963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:34.964013 9963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:34.973627 9971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:34.973671 9969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:34.973627 9968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:34.974001 9963 server_base.cc:1034] running on GCE node
I20260927 10:44:34.974385 9963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:34.975040 9963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:34.976197 9963 hybrid_clock.cc:648] HybridClock initialized: now 1790505874976165 us; error 44 us; skew 500 ppm
I20260927 10:44:34.977974 9963 webserver.cc:460] Webserver started at http://127.0.98.3:41219/ using document root <none> and password file <none>
I20260927 10:44:34.978586 9963 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:34.978648 9963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:34.978904 9963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:34.980979 9963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "869c38b2d88b487eb2b8df8d425765ef"
format_stamp: "Formatted at 2026-09-27 10:44:34 on dist-test-slave-dzhw"
server_key: "d13fe9273f04b4f971566bf6539e8450"
server_key_iv: "c2ec709e779a750e8d77b00f6581f982"
server_key_version: "encryptionkey@0"
I20260927 10:44:34.981519 9963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "869c38b2d88b487eb2b8df8d425765ef"
format_stamp: "Formatted at 2026-09-27 10:44:34 on dist-test-slave-dzhw"
server_key: "d13fe9273f04b4f971566bf6539e8450"
server_key_iv: "c2ec709e779a750e8d77b00f6581f982"
server_key_version: "encryptionkey@0"
I20260927 10:44:34.985057 9963 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260927 10:44:34.987527 9977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:34.988770 9963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:44:34.988897 9963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "869c38b2d88b487eb2b8df8d425765ef"
format_stamp: "Formatted at 2026-09-27 10:44:34 on dist-test-slave-dzhw"
server_key: "d13fe9273f04b4f971566bf6539e8450"
server_key_iv: "c2ec709e779a750e8d77b00f6581f982"
server_key_version: "encryptionkey@0"
I20260927 10:44:34.989027 9963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:35.008635 9963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:35.009358 9963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:35.009560 9963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:35.010219 9963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:35.011175 9963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:35.011220 9963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:35.011258 9963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:35.011274 9963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:35.019846 9963 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40165
I20260927 10:44:35.019868 10090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40165 every 8 connection(s)
I20260927 10:44:35.020783 9963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:44:35.025147 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 9963
I20260927 10:44:35.025256 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:44:35.025571 392 external_mini_cluster.cc:1428] Setting key fb15c30d152e9ed35b7c41dc79b4ae7a
I20260927 10:44:35.030089 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:35.022609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56525 (local address 127.0.98.62:41307)
0927 10:44:35.022740 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:35.022744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:35.023733 (+ 989us) server_negotiation.cc:407] Connection header received
0927 10:44:35.025199 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:35.025202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:35.025288 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:35.025387 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:35.026517 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:35.027156 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:35.028357 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:35.028568 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:35.028648 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:35.029704 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:35.029707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:35.029709 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:44:35.029719 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:35.029736 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:44:35.029743 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:35.029798 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:35.029930 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 10:44:35.029979 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:44:35.030987 10093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:35.022978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41307 (local address 127.0.98.3:56525)
0927 10:44:35.023592 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:35.023630 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:35.024958 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:35.025564 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:35.025573 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:35.025679 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:35.026315 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:35.026352 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:35.027279 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:35.027282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:35.028250 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:35.028258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:35.028698 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:35.028701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:35.028833 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:35.029458 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:44:35.029460 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:35.029500 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:44:35.029565 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:35.029814 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:35.029820 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:35.029824 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 10:44:35.029917 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 10:44:35.030139 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":109,"threads_started":1}
I20260927 10:44:35.031975 10091 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41307
I20260927 10:44:35.032219 10091 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:35.032722 10091 heartbeater.cc:511] Master 127.0.98.62:41307 requested a full tablet report, sending...
I20260927 10:44:35.033674 9640 ts_manager.cc:194] Registered new tserver with Master: 869c38b2d88b487eb2b8df8d425765ef (127.0.98.3:40165)
I20260927 10:44:35.034221 9640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:56525
I20260927 10:44:35.039697 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 10:44:35.042358 9680 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:35.041528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48018 (local address 127.0.98.62:41307)
0927 10:44:35.041695 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:35.041699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:35.041774 (+ 75us) server_negotiation.cc:407] Connection header received
0927 10:44:35.041872 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:35.041874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:35.041913 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:35.041976 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:35.042200 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48018: BlockingRecv error: recv got EOF from 127.0.0.1:48018 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 10:44:35.042480 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 9701
I20260927 10:44:35.047989 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 9832
I20260927 10:44:35.053259 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 9963
I20260927 10:44:35.058267 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 9610
2026-09-27T10:44:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (869 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T10:44:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:44:35Z Disabled control of system clock
I20260927 10:44:35.086901 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45865
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 10:44:35.197829 10102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:35.198108 10102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:35.198148 10102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:35.202277 10102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:44:35.202338 10102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:35.202354 10102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:35.202371 10102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:44:35.202450 10102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:44:35.207223 10102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45865
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:35.208369 10102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:35.209800 10102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:35.218631 10107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:35.218783 10108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:35.218672 10102 server_base.cc:1034] running on GCE node
W20260927 10:44:35.218734 10110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:35.219560 10102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:35.220690 10102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:35.221882 10102 hybrid_clock.cc:648] HybridClock initialized: now 1790505875221877 us; error 37 us; skew 500 ppm
I20260927 10:44:35.223826 10102 webserver.cc:460] Webserver started at http://127.0.98.62:36087/ using document root <none> and password file <none>
I20260927 10:44:35.224428 10102 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:35.224524 10102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:35.224759 10102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:35.226898 10102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "c3ca637dbe9e493894766102949a942f"
format_stamp: "Formatted at 2026-09-27 10:44:35 on dist-test-slave-dzhw"
server_key: "89bb63280716357b74cd607fd323015c"
server_key_iv: "1c0191032834706118aebf81c328e184"
server_key_version: "encryptionkey@0"
I20260927 10:44:35.227463 10102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "c3ca637dbe9e493894766102949a942f"
format_stamp: "Formatted at 2026-09-27 10:44:35 on dist-test-slave-dzhw"
server_key: "89bb63280716357b74cd607fd323015c"
server_key_iv: "1c0191032834706118aebf81c328e184"
server_key_version: "encryptionkey@0"
I20260927 10:44:35.231225 10102 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:44:35.233721 10116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:35.235082 10102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:44:35.235226 10102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "c3ca637dbe9e493894766102949a942f"
format_stamp: "Formatted at 2026-09-27 10:44:35 on dist-test-slave-dzhw"
server_key: "89bb63280716357b74cd607fd323015c"
server_key_iv: "1c0191032834706118aebf81c328e184"
server_key_version: "encryptionkey@0"
I20260927 10:44:35.235350 10102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:35.239923 10102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:35.240540 10102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:35.240721 10102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:35.247380 10102 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45865
I20260927 10:44:35.247390 10168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45865 every 8 connection(s)
I20260927 10:44:35.248387 10102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:44:35.251534 10169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:35.252712 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 10102
I20260927 10:44:35.252830 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:44:35.253134 392 external_mini_cluster.cc:1428] Setting key a39149022d3c1f515ee74a55f9092b76
I20260927 10:44:35.257905 10169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3ca637dbe9e493894766102949a942f: Bootstrap starting.
I20260927 10:44:35.258746 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:35.254621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38010 (local address 127.0.98.62:45865)
0927 10:44:35.255291 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:35.255307 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:35.255342 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:44:35.255956 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:35.255972 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:35.256305 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:35.256637 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:35.257400 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:35.257405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:35.257416 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:44:35.257426 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:35.257477 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:44:35.257492 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:35.257643 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:35.257711 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 10:44:35.257963 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":143,"threads_started":1}
I20260927 10:44:35.260360 10169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3ca637dbe9e493894766102949a942f: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:35.261200 10169 log.cc:824] T 00000000000000000000000000000000 P c3ca637dbe9e493894766102949a942f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:35.263029 10169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3ca637dbe9e493894766102949a942f: No bootstrap required, opened a new log
I20260927 10:44:35.266722 10169 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3ca637dbe9e493894766102949a942f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3ca637dbe9e493894766102949a942f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45865 } }
I20260927 10:44:35.266981 10169 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3ca637dbe9e493894766102949a942f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:35.267016 10169 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3ca637dbe9e493894766102949a942f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3ca637dbe9e493894766102949a942f, State: Initialized, Role: FOLLOWER
I20260927 10:44:35.267406 10169 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3ca637dbe9e493894766102949a942f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3ca637dbe9e493894766102949a942f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45865 } }
I20260927 10:44:35.267524 10169 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3ca637dbe9e493894766102949a942f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:44:35.267568 10169 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3ca637dbe9e493894766102949a942f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:44:35.267637 10169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3ca637dbe9e493894766102949a942f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:35.268463 10169 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3ca637dbe9e493894766102949a942f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3ca637dbe9e493894766102949a942f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45865 } }
I20260927 10:44:35.268740 10169 leader_election.cc:302] T 00000000000000000000000000000000 P c3ca637dbe9e493894766102949a942f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3ca637dbe9e493894766102949a942f; no voters:
I20260927 10:44:35.269001 10169 leader_election.cc:288] T 00000000000000000000000000000000 P c3ca637dbe9e493894766102949a942f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:44:35.269129 10174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3ca637dbe9e493894766102949a942f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:35.269387 10174 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3ca637dbe9e493894766102949a942f [term 1 LEADER]: Becoming Leader. State: Replica: c3ca637dbe9e493894766102949a942f, State: Running, Role: LEADER
I20260927 10:44:35.269723 10174 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3ca637dbe9e493894766102949a942f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3ca637dbe9e493894766102949a942f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45865 } }
I20260927 10:44:35.270207 10169 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3ca637dbe9e493894766102949a942f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:44:35.271394 10176 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3ca637dbe9e493894766102949a942f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3ca637dbe9e493894766102949a942f. Latest consensus state: current_term: 1 leader_uuid: "c3ca637dbe9e493894766102949a942f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3ca637dbe9e493894766102949a942f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45865 } } }
I20260927 10:44:35.271430 10175 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3ca637dbe9e493894766102949a942f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3ca637dbe9e493894766102949a942f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3ca637dbe9e493894766102949a942f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45865 } } }
I20260927 10:44:35.271555 10176 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3ca637dbe9e493894766102949a942f [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:35.271647 10175 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3ca637dbe9e493894766102949a942f [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:35.272038 10184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:44:35.275089 10184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:44:35.280525 10184 catalog_manager.cc:1347] Generated new cluster ID: 04c95e76f8bd40e19bb2d2b87d269e87
I20260927 10:44:35.280611 10184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:44:35.297150 10184 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:44:35.297506 10184 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:44:35.310112 10184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3ca637dbe9e493894766102949a942f: Generated new TSK 0
I20260927 10:44:35.311028 10184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:44:35.317167 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45865
--builtin_ntp_servers=127.0.98.20:37847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 10:44:35.431669 10193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:35.432129 10193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:35.432454 10193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:35.437486 10193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:35.437556 10193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:35.437616 10193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:44:35.443037 10193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:35.444195 10193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:35.445621 10193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:35.455394 10198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:35.455394 10199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:35.455404 10201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:35.455549 10193 server_base.cc:1034] running on GCE node
I20260927 10:44:35.456113 10193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:35.456764 10193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:35.457919 10193 hybrid_clock.cc:648] HybridClock initialized: now 1790505875457903 us; error 31 us; skew 500 ppm
I20260927 10:44:35.459800 10193 webserver.cc:460] Webserver started at http://127.0.98.1:42233/ using document root <none> and password file <none>
I20260927 10:44:35.460378 10193 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:35.460438 10193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:35.460650 10193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:35.462760 10193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "bb5d105694904bceae6dec82c5253e05"
format_stamp: "Formatted at 2026-09-27 10:44:35 on dist-test-slave-dzhw"
server_key: "69b62f223d8aad816da142efe7da00cd"
server_key_iv: "38491e6224ce758b7ff3382ab54353ea"
server_key_version: "encryptionkey@0"
I20260927 10:44:35.463320 10193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bb5d105694904bceae6dec82c5253e05"
format_stamp: "Formatted at 2026-09-27 10:44:35 on dist-test-slave-dzhw"
server_key: "69b62f223d8aad816da142efe7da00cd"
server_key_iv: "38491e6224ce758b7ff3382ab54353ea"
server_key_version: "encryptionkey@0"
I20260927 10:44:35.467033 10193 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:44:35.469470 10207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:35.470808 10193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:44:35.470953 10193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "bb5d105694904bceae6dec82c5253e05"
format_stamp: "Formatted at 2026-09-27 10:44:35 on dist-test-slave-dzhw"
server_key: "69b62f223d8aad816da142efe7da00cd"
server_key_iv: "38491e6224ce758b7ff3382ab54353ea"
server_key_version: "encryptionkey@0"
I20260927 10:44:35.471086 10193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:35.475803 10193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:35.476410 10193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:35.476598 10193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:35.477125 10193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:35.478040 10193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:35.478109 10193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:35.478179 10193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:35.478231 10193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:35.486809 10193 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40783
I20260927 10:44:35.486845 10320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40783 every 8 connection(s)
I20260927 10:44:35.487771 10193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:44:35.493669 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:35.489593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50141 (local address 127.0.98.62:45865)
0927 10:44:35.489727 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:35.489731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:35.490809 (+ 1078us) server_negotiation.cc:407] Connection header received
0927 10:44:35.492086 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:35.492089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:35.492130 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:35.492223 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:35.493236 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:35.493239 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:35.493241 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:44:35.493252 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:35.493270 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:44:35.493278 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:35.493347 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:35.493455 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 10:44:35.493503 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:44:35.493681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 10193
I20260927 10:44:35.493783 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:44:35.494055 392 external_mini_cluster.cc:1428] Setting key 439c050817a087ab478b68c5cdf02ae7
I20260927 10:44:35.494697 10323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:35.489986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45865 (local address 127.0.98.1:50141)
0927 10:44:35.490670 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:35.490710 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:35.491887 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:35.492369 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:35.492376 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:35.492460 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:35.492987 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:44:35.492989 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:35.493034 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:44:35.493100 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:35.493338 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:35.493342 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:35.493346 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 10:44:35.493462 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 10:44:35.493655 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":105,"threads_started":1}
I20260927 10:44:35.495887 10321 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45865
I20260927 10:44:35.496069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45865
--builtin_ntp_servers=127.0.98.20:37847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 10:44:35.496135 10321 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:35.496588 10321 heartbeater.cc:511] Master 127.0.98.62:45865 requested a full tablet report, sending...
I20260927 10:44:35.498313 10133 ts_manager.cc:194] Registered new tserver with Master: bb5d105694904bceae6dec82c5253e05 (127.0.98.1:40783)
W20260927 10:44:35.607196 10324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:35.607474 10324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:35.607514 10324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:35.611557 10324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:35.611620 10324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:35.611676 10324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:44:35.616309 10324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:35.617326 10324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:35.618707 10324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:35.628073 10330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:35.628072 10329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:35.628065 10332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:35.628196 10324 server_base.cc:1034] running on GCE node
I20260927 10:44:35.628743 10324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:35.629406 10324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:35.630587 10324 hybrid_clock.cc:648] HybridClock initialized: now 1790505875630564 us; error 34 us; skew 500 ppm
I20260927 10:44:35.632571 10324 webserver.cc:460] Webserver started at http://127.0.98.2:40135/ using document root <none> and password file <none>
I20260927 10:44:35.633168 10324 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:35.633262 10324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:35.633492 10324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:35.635639 10324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "9fb9e4ce99e84f71b55fbb5d85d0d684"
format_stamp: "Formatted at 2026-09-27 10:44:35 on dist-test-slave-dzhw"
server_key: "d29b7ac0c15c9fe9c243f46f9ff3b97e"
server_key_iv: "af72961fb0949aa33dad75f16bd55cda"
server_key_version: "encryptionkey@0"
I20260927 10:44:35.636202 10324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9fb9e4ce99e84f71b55fbb5d85d0d684"
format_stamp: "Formatted at 2026-09-27 10:44:35 on dist-test-slave-dzhw"
server_key: "d29b7ac0c15c9fe9c243f46f9ff3b97e"
server_key_iv: "af72961fb0949aa33dad75f16bd55cda"
server_key_version: "encryptionkey@0"
I20260927 10:44:35.640021 10324 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:44:35.642561 10338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:35.643787 10324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:44:35.643930 10324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "9fb9e4ce99e84f71b55fbb5d85d0d684"
format_stamp: "Formatted at 2026-09-27 10:44:35 on dist-test-slave-dzhw"
server_key: "d29b7ac0c15c9fe9c243f46f9ff3b97e"
server_key_iv: "af72961fb0949aa33dad75f16bd55cda"
server_key_version: "encryptionkey@0"
I20260927 10:44:35.644052 10324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:35.648773 10324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:35.649386 10324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:35.649582 10324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:35.650130 10324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:35.651121 10324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:35.651196 10324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:35.651260 10324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:35.651306 10324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:35.661325 10324 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46197
I20260927 10:44:35.661346 10451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46197 every 8 connection(s)
I20260927 10:44:35.662225 10324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:44:35.662971 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 10324
I20260927 10:44:35.663100 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:44:35.663360 392 external_mini_cluster.cc:1428] Setting key f8b150eaeb76b5c3e869de45b5d99354
I20260927 10:44:35.665490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45865
--builtin_ntp_servers=127.0.98.20:37847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 10:44:35.669293 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:35.664325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51253 (local address 127.0.98.62:45865)
0927 10:44:35.664460 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:35.664464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:35.665806 (+ 1342us) server_negotiation.cc:407] Connection header received
0927 10:44:35.667488 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:35.667491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:35.667527 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:35.667601 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:35.668855 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:35.668858 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:35.668860 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:44:35.668871 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:35.668889 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:44:35.668897 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:35.668984 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:35.669144 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 10:44:35.669182 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:44:35.670568 10454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:35.664888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45865 (local address 127.0.98.2:51253)
0927 10:44:35.665633 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:35.665698 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:35.667238 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:35.667822 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:35.667832 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:35.667965 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:35.668546 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:44:35.668550 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:35.668605 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:44:35.668694 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:35.668991 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:35.669003 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:35.669012 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 10:44:35.669188 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 10:44:35.669451 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":131,"threads_started":1}
I20260927 10:44:35.671859 10452 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45865
I20260927 10:44:35.672106 10452 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:35.672538 10452 heartbeater.cc:511] Master 127.0.98.62:45865 requested a full tablet report, sending...
I20260927 10:44:35.673673 10133 ts_manager.cc:194] Registered new tserver with Master: 9fb9e4ce99e84f71b55fbb5d85d0d684 (127.0.98.2:46197)
W20260927 10:44:35.780207 10455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:35.780516 10455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:35.780587 10455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:35.784714 10455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:35.784806 10455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:35.784902 10455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:44:35.789834 10455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:35.791134 10455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:35.792544 10455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:35.801986 10461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:35.801990 10460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:35.802142 10455 server_base.cc:1034] running on GCE node
W20260927 10:44:35.802336 10463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:35.802801 10455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:35.803445 10455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:35.804631 10455 hybrid_clock.cc:648] HybridClock initialized: now 1790505875804617 us; error 43 us; skew 500 ppm
I20260927 10:44:35.806391 10455 webserver.cc:460] Webserver started at http://127.0.98.3:41313/ using document root <none> and password file <none>
I20260927 10:44:35.807013 10455 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:35.807106 10455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:35.807312 10455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:35.809305 10455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "d8d734afd13b4695bd7120c66d6240ad"
format_stamp: "Formatted at 2026-09-27 10:44:35 on dist-test-slave-dzhw"
server_key: "dd10f0879cf268175d8066ce321cc487"
server_key_iv: "e8077ba89ad9c7bb620360ffa8d07fbb"
server_key_version: "encryptionkey@0"
I20260927 10:44:35.809885 10455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d8d734afd13b4695bd7120c66d6240ad"
format_stamp: "Formatted at 2026-09-27 10:44:35 on dist-test-slave-dzhw"
server_key: "dd10f0879cf268175d8066ce321cc487"
server_key_iv: "e8077ba89ad9c7bb620360ffa8d07fbb"
server_key_version: "encryptionkey@0"
I20260927 10:44:35.813503 10455 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:44:35.815995 10469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:35.817322 10455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:44:35.817487 10455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "d8d734afd13b4695bd7120c66d6240ad"
format_stamp: "Formatted at 2026-09-27 10:44:35 on dist-test-slave-dzhw"
server_key: "dd10f0879cf268175d8066ce321cc487"
server_key_iv: "e8077ba89ad9c7bb620360ffa8d07fbb"
server_key_version: "encryptionkey@0"
I20260927 10:44:35.817618 10455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:35.822279 10455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:35.822930 10455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:35.823113 10455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:35.823652 10455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:35.824564 10455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:35.824633 10455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:35.824698 10455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:35.824752 10455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:35.833595 10455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36151
I20260927 10:44:35.833618 10582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36151 every 8 connection(s)
I20260927 10:44:35.834553 10455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:44:35.840246 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:35.836286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43631 (local address 127.0.98.62:45865)
0927 10:44:35.836451 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:35.836456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:35.837471 (+ 1015us) server_negotiation.cc:407] Connection header received
0927 10:44:35.838753 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:35.838755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:35.838791 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:35.838865 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:35.839877 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:35.839880 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:35.839882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:44:35.839892 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:35.839910 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:44:35.839918 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:35.839992 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:35.840107 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 10:44:35.840144 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:44:35.841081 10585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:35.836702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45865 (local address 127.0.98.3:43631)
0927 10:44:35.837323 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:35.837362 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:35.838576 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:35.839017 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:35.839023 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:35.839107 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:35.839642 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:44:35.839643 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:35.839681 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:44:35.839745 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:35.839995 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:35.840001 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:35.840005 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 10:44:35.840129 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 10:44:35.840334 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":106,"threads_started":1}
I20260927 10:44:35.842104 10583 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45865
I20260927 10:44:35.842329 10583 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:35.842679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 10455
I20260927 10:44:35.842784 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:44:35.842810 10583 heartbeater.cc:511] Master 127.0.98.62:45865 requested a full tablet report, sending...
I20260927 10:44:35.843080 392 external_mini_cluster.cc:1428] Setting key f73adaadb6d8423d77aa4ce41836eead
I20260927 10:44:35.843977 10133 ts_manager.cc:194] Registered new tserver with Master: d8d734afd13b4695bd7120c66d6240ad (127.0.98.3:36151)
I20260927 10:44:35.845117 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 10:44:35.848323 10172 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:44:35.847321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38026 (local address 127.0.98.62:45865)
0927 10:44:35.847464 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:35.847469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:35.847664 (+ 195us) server_negotiation.cc:407] Connection header received
0927 10:44:35.847843 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:35.847847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:35.847879 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:35.847971 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:35.848177 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38026: BlockingRecv error: recv got EOF from 127.0.0.1:38026 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:44:35.848667 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 10193
I20260927 10:44:35.854285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 10324
I20260927 10:44:35.859979 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 10455
I20260927 10:44:35.865175 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 10102
2026-09-27T10:44:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (806 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:44:35 dist-test-slave-dzhw krb5kdc[10593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:44:35 dist-test-slave-dzhw krb5kdc[10593](info): set up 2 sockets
Sep 27 10:44:35 dist-test-slave-dzhw krb5kdc[10593](info): commencing operation
krb5kdc: starting...
W20260927 10:44:36.907397 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:44:36 dist-test-slave-dzhw krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:44:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:44:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:44:37.026232 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38335
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:37.133486 10606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:37.133764 10606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:37.133803 10606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:37.137790 10606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:44:37.137861 10606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:37.137880 10606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:37.137898 10606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:44:37.137913 10606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:44:37.143328 10606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:37.144407 10606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:37.145416 10606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:37.153846 10611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:37.153821 10612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:37.153827 10614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:37.154178 10606 server_base.cc:1034] running on GCE node
I20260927 10:44:37.154778 10606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:37.155800 10606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:37.156989 10606 hybrid_clock.cc:648] HybridClock initialized: now 1790505877156982 us; error 30 us; skew 500 ppm
Sep 27 10:44:37 dist-test-slave-dzhw krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:37.166116 10606 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:44:37.167125 10606 webserver.cc:460] Webserver started at http://127.0.98.62:34155/ using document root <none> and password file <none>
I20260927 10:44:37.167703 10606 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:37.167793 10606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:37.168031 10606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:37.170055 10606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "2533753efdc94fa4b81d4bc449ef2a4d"
format_stamp: "Formatted at 2026-09-27 10:44:37 on dist-test-slave-dzhw"
server_key: "1df13ff3493eec1476dae19cf565889b"
server_key_iv: "ca02d4ff6d72194f4b1254f9b7649fc7"
server_key_version: "encryptionkey@0"
I20260927 10:44:37.170661 10606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "2533753efdc94fa4b81d4bc449ef2a4d"
format_stamp: "Formatted at 2026-09-27 10:44:37 on dist-test-slave-dzhw"
server_key: "1df13ff3493eec1476dae19cf565889b"
server_key_iv: "ca02d4ff6d72194f4b1254f9b7649fc7"
server_key_version: "encryptionkey@0"
I20260927 10:44:37.174192 10606 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:44:37.176515 10621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:37.177757 10606 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:44:37.177882 10606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "2533753efdc94fa4b81d4bc449ef2a4d"
format_stamp: "Formatted at 2026-09-27 10:44:37 on dist-test-slave-dzhw"
server_key: "1df13ff3493eec1476dae19cf565889b"
server_key_iv: "ca02d4ff6d72194f4b1254f9b7649fc7"
server_key_version: "encryptionkey@0"
I20260927 10:44:37.178007 10606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:37.195077 10606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:37.199704 10606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:37.199913 10606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:37.207866 10606 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38335
I20260927 10:44:37.208088 10673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38335 every 8 connection(s)
I20260927 10:44:37.208880 10606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:44:37.211632 10674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:37.212560 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 10606
I20260927 10:44:37.212697 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:44:37.212967 392 external_mini_cluster.cc:1428] Setting key 37db15d96314c63e5cf0cbb6df4fa2b1
I20260927 10:44:37.217308 10674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2533753efdc94fa4b81d4bc449ef2a4d: Bootstrap starting.
I20260927 10:44:37.219488 10674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2533753efdc94fa4b81d4bc449ef2a4d: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:37.220321 10674 log.cc:824] T 00000000000000000000000000000000 P 2533753efdc94fa4b81d4bc449ef2a4d: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:44:37 dist-test-slave-dzhw krb5kdc[10593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:44:37.222591 10674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2533753efdc94fa4b81d4bc449ef2a4d: No bootstrap required, opened a new log
I20260927 10:44:37.226598 10674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2533753efdc94fa4b81d4bc449ef2a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2533753efdc94fa4b81d4bc449ef2a4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38335 } }
I20260927 10:44:37.226773 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:37.214323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33516 (local address 127.0.98.62:38335)
0927 10:44:37.214767 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:37.214782 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:37.214815 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:44:37.215427 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:37.215442 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:37.215760 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:37.216126 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:37.216903 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.217913 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.218810 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.219266 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.219446 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:37.222210 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:37.222215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:37.222228 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:37.222252 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:37.223923 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:37.224443 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:37.224448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:37.224452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:37.224556 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:37.224844 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:37.224846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:37.224847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:37.225222 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:37.225359 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:37.225650 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 10:44:37.225952 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":138,"threads_started":1}
I20260927 10:44:37.226806 10674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2533753efdc94fa4b81d4bc449ef2a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:37.226928 10674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2533753efdc94fa4b81d4bc449ef2a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2533753efdc94fa4b81d4bc449ef2a4d, State: Initialized, Role: FOLLOWER
I20260927 10:44:37.227383 10674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2533753efdc94fa4b81d4bc449ef2a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2533753efdc94fa4b81d4bc449ef2a4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38335 } }
I20260927 10:44:37.227504 10674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2533753efdc94fa4b81d4bc449ef2a4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:44:37.227584 10674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2533753efdc94fa4b81d4bc449ef2a4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:44:37.227681 10674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2533753efdc94fa4b81d4bc449ef2a4d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:37.228775 10674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2533753efdc94fa4b81d4bc449ef2a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2533753efdc94fa4b81d4bc449ef2a4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38335 } }
I20260927 10:44:37.229106 10674 leader_election.cc:302] T 00000000000000000000000000000000 P 2533753efdc94fa4b81d4bc449ef2a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2533753efdc94fa4b81d4bc449ef2a4d; no voters:
I20260927 10:44:37.229434 10674 leader_election.cc:288] T 00000000000000000000000000000000 P 2533753efdc94fa4b81d4bc449ef2a4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:44:37.229557 10679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2533753efdc94fa4b81d4bc449ef2a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:37.229821 10679 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2533753efdc94fa4b81d4bc449ef2a4d [term 1 LEADER]: Becoming Leader. State: Replica: 2533753efdc94fa4b81d4bc449ef2a4d, State: Running, Role: LEADER
I20260927 10:44:37.230114 10679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2533753efdc94fa4b81d4bc449ef2a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2533753efdc94fa4b81d4bc449ef2a4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38335 } }
I20260927 10:44:37.230659 10674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2533753efdc94fa4b81d4bc449ef2a4d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:44:37.231810 10681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2533753efdc94fa4b81d4bc449ef2a4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2533753efdc94fa4b81d4bc449ef2a4d. Latest consensus state: current_term: 1 leader_uuid: "2533753efdc94fa4b81d4bc449ef2a4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2533753efdc94fa4b81d4bc449ef2a4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38335 } } }
I20260927 10:44:37.231812 10680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2533753efdc94fa4b81d4bc449ef2a4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2533753efdc94fa4b81d4bc449ef2a4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2533753efdc94fa4b81d4bc449ef2a4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38335 } } }
I20260927 10:44:37.231962 10681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2533753efdc94fa4b81d4bc449ef2a4d [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:37.231967 10680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2533753efdc94fa4b81d4bc449ef2a4d [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:37.232563 10688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:44:37.235502 10688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:44:37.240518 10688 catalog_manager.cc:1347] Generated new cluster ID: 64c20ca3616943f4b6b7d7a972555114
I20260927 10:44:37.240593 10688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:44:37.254045 10688 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:44:37.255100 10688 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:44:37.264499 10688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2533753efdc94fa4b81d4bc449ef2a4d: Generated new TSK 0
I20260927 10:44:37.265281 10688 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 10:44:37.297266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38335
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:37.409756 10700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:37.410040 10700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:37.410084 10700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:37.414263 10700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:37.414335 10700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:37.414427 10700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:44:37.419870 10700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:37.420989 10700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:37.422021 10700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:44:37.431512 10700 server_base.cc:1034] running on GCE node
W20260927 10:44:37.431532 10708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:37.431485 10706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:37.431486 10705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:37.432303 10700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:37.432965 10700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:37.434155 10700 hybrid_clock.cc:648] HybridClock initialized: now 1790505877434127 us; error 49 us; skew 500 ppm
Sep 27 10:44:37 dist-test-slave-dzhw krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:37.437091 10700 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 10:44:37.437919 10700 webserver.cc:460] Webserver started at http://127.0.98.1:35807/ using document root <none> and password file <none>
I20260927 10:44:37.438534 10700 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:37.438602 10700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:37.438776 10700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:37.440759 10700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "dd0f89628d57420e99f893bf5140052d"
format_stamp: "Formatted at 2026-09-27 10:44:37 on dist-test-slave-dzhw"
server_key: "7706ea92771e3e5907635c9f005cabff"
server_key_iv: "480efd435d1052ba23def7eb44da605d"
server_key_version: "encryptionkey@0"
I20260927 10:44:37.441270 10700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dd0f89628d57420e99f893bf5140052d"
format_stamp: "Formatted at 2026-09-27 10:44:37 on dist-test-slave-dzhw"
server_key: "7706ea92771e3e5907635c9f005cabff"
server_key_iv: "480efd435d1052ba23def7eb44da605d"
server_key_version: "encryptionkey@0"
I20260927 10:44:37.445037 10700 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 10:44:37.447578 10715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:37.448839 10700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:44:37.448992 10700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "dd0f89628d57420e99f893bf5140052d"
format_stamp: "Formatted at 2026-09-27 10:44:37 on dist-test-slave-dzhw"
server_key: "7706ea92771e3e5907635c9f005cabff"
server_key_iv: "480efd435d1052ba23def7eb44da605d"
server_key_version: "encryptionkey@0"
I20260927 10:44:37.449107 10700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:37.469357 10700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:37.472544 10700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:37.472751 10700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:37.473366 10700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:37.474172 10700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:37.474246 10700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:37.474313 10700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:37.474362 10700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:37.482373 10700 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36685
I20260927 10:44:37.482462 10828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36685 every 8 connection(s)
I20260927 10:44:37.483355 10700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:44:37.483613 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 10700
I20260927 10:44:37.483736 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:44:37.484073 392 external_mini_cluster.cc:1428] Setting key 5d2cc0b85d3414732d4976b52a7681d5
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 27 10:44:37 dist-test-slave-dzhw krb5kdc[10593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 10:44:37.498781 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:37.485291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50675 (local address 127.0.98.62:38335)
0927 10:44:37.485450 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:37.485454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:37.486746 (+ 1292us) server_negotiation.cc:407] Connection header received
0927 10:44:37.487901 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:37.487907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:37.487968 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:37.488071 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:37.489507 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.490127 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.492014 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.492234 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.492315 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:37.495195 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:37.495198 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:37.495200 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:37.495223 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:37.496613 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:37.497142 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:37.497145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:37.497147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:37.497237 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:37.497662 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:37.497666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:37.497667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:37.497849 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:37.497956 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:37.498480 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 10:44:37.498609 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:44:37.499542 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:37.485761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38335 (local address 127.0.98.1:50675)
0927 10:44:37.486563 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:37.486620 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:37.487694 (+ 1074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:37.488207 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:37.488213 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:37.488671 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:37.489344 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:37.489366 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.490265 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.490271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:37.491889 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:37.491904 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.492398 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.492402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:37.492623 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:37.493240 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:37.493262 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:37.495038 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:37.496718 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:37.496722 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:37.496735 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:37.497002 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:37.497353 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:37.497359 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:37.497364 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:37.497542 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:37.497954 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:37.497961 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:37.498208 (+ 247us) client_negotiation.cc:769] Sending connection context
0927 10:44:37.498390 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 10:44:37.498728 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":111,"threads_started":1}
I20260927 10:44:37.500752 10829 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38335
I20260927 10:44:37.501037 10829 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:37.501624 10829 heartbeater.cc:511] Master 127.0.98.62:38335 requested a full tablet report, sending...
I20260927 10:44:37.503188 10633 ts_manager.cc:194] Registered new tserver with Master: dd0f89628d57420e99f893bf5140052d (127.0.98.1:36685)
I20260927 10:44:37.504264 10633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 10:44:37.513734 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38335
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:37.622598 10834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:37.622861 10834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:37.622905 10834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:37.626931 10834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:37.626995 10834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:37.627111 10834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:44:37.632535 10834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:37.633577 10834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:37.634651 10834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:37.644196 10839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:37.644248 10842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:37.644299 10834 server_base.cc:1034] running on GCE node
W20260927 10:44:37.644204 10840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:37.644865 10834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:37.645547 10834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:37.646739 10834 hybrid_clock.cc:648] HybridClock initialized: now 1790505877646713 us; error 41 us; skew 500 ppm
Sep 27 10:44:37 dist-test-slave-dzhw krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:37.649420 10834 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 10:44:37.650270 10834 webserver.cc:460] Webserver started at http://127.0.98.2:39359/ using document root <none> and password file <none>
I20260927 10:44:37.650866 10834 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:37.650954 10834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:37.651190 10834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:37.653354 10834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "08c6f86a2dac4810aa8571683b0932b0"
format_stamp: "Formatted at 2026-09-27 10:44:37 on dist-test-slave-dzhw"
server_key: "1d3b12d874ac8208b39f988ff235e598"
server_key_iv: "4600d5b8d85e599d1857c131d2b44687"
server_key_version: "encryptionkey@0"
I20260927 10:44:37.654021 10834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "08c6f86a2dac4810aa8571683b0932b0"
format_stamp: "Formatted at 2026-09-27 10:44:37 on dist-test-slave-dzhw"
server_key: "1d3b12d874ac8208b39f988ff235e598"
server_key_iv: "4600d5b8d85e599d1857c131d2b44687"
server_key_version: "encryptionkey@0"
I20260927 10:44:37.657802 10834 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:44:37.660390 10849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:37.661599 10834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:44:37.661734 10834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "08c6f86a2dac4810aa8571683b0932b0"
format_stamp: "Formatted at 2026-09-27 10:44:37 on dist-test-slave-dzhw"
server_key: "1d3b12d874ac8208b39f988ff235e598"
server_key_iv: "4600d5b8d85e599d1857c131d2b44687"
server_key_version: "encryptionkey@0"
I20260927 10:44:37.661845 10834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:37.674727 10834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:37.677874 10834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:37.678082 10834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:37.678701 10834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:37.679564 10834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:37.679628 10834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:37.679692 10834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:37.679738 10834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:37.688931 10834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44223
I20260927 10:44:37.688957 10962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44223 every 8 connection(s)
I20260927 10:44:37.689860 10834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 10:44:37 dist-test-slave-dzhw krb5kdc[10593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:44:37.699975 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 10834
I20260927 10:44:37.700138 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:44:37.700513 392 external_mini_cluster.cc:1428] Setting key 371138f25e86a82299b5b2a5d81fcfb2
I20260927 10:44:37.703943 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:37.691717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44309 (local address 127.0.98.62:38335)
0927 10:44:37.691847 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:37.691850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:37.692874 (+ 1024us) server_negotiation.cc:407] Connection header received
0927 10:44:37.693769 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:37.693773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:37.693824 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:37.693936 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:37.695161 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.695679 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.696801 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.697014 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.697132 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:37.699742 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:37.699746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:37.699751 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:37.699787 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:37.701400 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:37.702079 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:37.702084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:37.702086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:37.702180 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:37.702636 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:37.702639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:37.702640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:37.702816 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:37.702908 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:37.703578 (+ 670us) server_negotiation.cc:299] Negotiation successful
0927 10:44:37.703743 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:44:37.704783 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:37.692094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38335 (local address 127.0.98.2:44309)
0927 10:44:37.692735 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:37.692773 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:37.693574 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:37.694108 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:37.694115 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:37.694544 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:37.695017 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:37.695029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.695803 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.695807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:37.696676 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:37.696685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.697171 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.697175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:37.697337 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:37.697920 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:37.697936 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:37.699543 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:37.701533 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:37.701538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:37.701551 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:37.701974 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:37.702284 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:37.702287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:37.702289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:37.702517 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:37.702929 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:37.702939 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:37.703284 (+ 345us) client_negotiation.cc:769] Sending connection context
0927 10:44:37.703492 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 10:44:37.703887 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":110,"threads_started":1}
I20260927 10:44:37.705956 10963 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38335
I20260927 10:44:37.706214 10963 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:37.706763 10963 heartbeater.cc:511] Master 127.0.98.62:38335 requested a full tablet report, sending...
I20260927 10:44:37.707849 10633 ts_manager.cc:194] Registered new tserver with Master: 08c6f86a2dac4810aa8571683b0932b0 (127.0.98.2:44223)
I20260927 10:44:37.708658 10633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44309
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 10:44:37.729517 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38335
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:37.840343 10968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:37.840600 10968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:37.840643 10968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:37.844691 10968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:37.844754 10968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:37.844842 10968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:44:37.850109 10968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:44:37.851223 10968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:37.852231 10968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:37.862123 10973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:37.862241 10968 server_base.cc:1034] running on GCE node
W20260927 10:44:37.862121 10976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:37.862175 10974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:37.862768 10968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:37.863399 10968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:37.864588 10968 hybrid_clock.cc:648] HybridClock initialized: now 1790505877864563 us; error 36 us; skew 500 ppm
Sep 27 10:44:37 dist-test-slave-dzhw krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:37.867486 10968 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 10:44:37.868358 10968 webserver.cc:460] Webserver started at http://127.0.98.3:35357/ using document root <none> and password file <none>
I20260927 10:44:37.868923 10968 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:37.869009 10968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:37.869230 10968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:37.871290 10968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "67f807034d2e43d591e068c05ac674c4"
format_stamp: "Formatted at 2026-09-27 10:44:37 on dist-test-slave-dzhw"
server_key: "69146a90be06956d53e445fe10957dd4"
server_key_iv: "3c7b1fb504a9faa94c2f9d73a0649c52"
server_key_version: "encryptionkey@0"
I20260927 10:44:37.871853 10968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "67f807034d2e43d591e068c05ac674c4"
format_stamp: "Formatted at 2026-09-27 10:44:37 on dist-test-slave-dzhw"
server_key: "69146a90be06956d53e445fe10957dd4"
server_key_iv: "3c7b1fb504a9faa94c2f9d73a0649c52"
server_key_version: "encryptionkey@0"
I20260927 10:44:37.875504 10968 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:44:37.877918 10983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:37.879278 10968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:44:37.879428 10968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "67f807034d2e43d591e068c05ac674c4"
format_stamp: "Formatted at 2026-09-27 10:44:37 on dist-test-slave-dzhw"
server_key: "69146a90be06956d53e445fe10957dd4"
server_key_iv: "3c7b1fb504a9faa94c2f9d73a0649c52"
server_key_version: "encryptionkey@0"
I20260927 10:44:37.879546 10968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:37.907335 10968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:37.910792 10968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:37.911005 10968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:37.911597 10968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:37.912417 10968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:37.912489 10968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:37.912559 10968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:37.912612 10968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:37.922184 10968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36565
I20260927 10:44:37.922240 11096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36565 every 8 connection(s)
I20260927 10:44:37.923430 10968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:44:37.926303 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 10968
I20260927 10:44:37.926455 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:44:37.926810 392 external_mini_cluster.cc:1428] Setting key 433e40ba942cbf4779ce6fd43abf57fe
Sep 27 10:44:37 dist-test-slave-dzhw krb5kdc[10593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:44:37.937469 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:37.925314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36571 (local address 127.0.98.62:38335)
0927 10:44:37.925476 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:37.925479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:37.926724 (+ 1245us) server_negotiation.cc:407] Connection header received
0927 10:44:37.927877 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:37.927880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:37.927949 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:37.928040 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:37.929369 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.929884 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.931167 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.931453 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.931533 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:37.933906 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:37.933909 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:37.933911 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:37.933934 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:37.935370 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:37.935925 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:37.935928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:37.935929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:37.936009 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:37.936358 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:37.936362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:37.936363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:37.936567 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:37.936646 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:37.937231 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 10:44:37.937340 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 10:44:37.938371 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:37.925712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38335 (local address 127.0.98.3:36571)
0927 10:44:37.926539 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:37.926598 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:37.927657 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:37.928265 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:37.928272 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:37.928664 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:37.929243 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:37.929257 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.929994 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.929997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:37.931059 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:37.931069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.931580 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.931582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:37.931718 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:37.932314 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:37.932330 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:37.933744 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:37.935491 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:37.935496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:37.935510 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:37.935793 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:37.936113 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:37.936116 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:37.936117 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:37.936256 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:37.936685 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:37.936691 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:37.936991 (+ 300us) client_negotiation.cc:769] Sending connection context
0927 10:44:37.937164 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 10:44:37.937436 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":119,"threads_started":1}
I20260927 10:44:37.939734 11097 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38335
I20260927 10:44:37.939993 11097 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:37.940522 11097 heartbeater.cc:511] Master 127.0.98.62:38335 requested a full tablet report, sending...
I20260927 10:44:37.941529 10633 ts_manager.cc:194] Registered new tserver with Master: 67f807034d2e43d591e068c05ac674c4 (127.0.98.3:36565)
I20260927 10:44:37.942091 10633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36571
I20260927 10:44:37.952498 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:44:37.961997 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:37.954520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33532 (local address 127.0.98.62:38335)
0927 10:44:37.954762 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:37.954768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:37.954972 (+ 204us) server_negotiation.cc:407] Connection header received
0927 10:44:37.955130 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:37.955133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:37.955179 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:37.955273 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:37.955989 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.956511 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.957363 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.957592 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.957700 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:37.959116 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:37.959120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:37.959122 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:37.959147 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:37.960336 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:37.960760 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:37.960765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:37.960766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:37.960849 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:37.961118 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:37.961122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:37.961124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:37.961345 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:37.961490 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:37.961726 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 10:44:37.961836 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260927 10:44:37.964802 10633 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:44:37.968031 10633 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:44:37.981906 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:37.976749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33186 (local address 127.0.98.2:44223)
0927 10:44:37.977288 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:37.977300 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:37.977320 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:44:37.977396 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:37.977402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:37.977622 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:37.977798 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:37.978151 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.978965 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.980597 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.981239 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.981360 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:37.981515 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:37.981668 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 10:44:37.981752 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":130,"threads_started":1}
I20260927 10:44:37.981909 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:37.976153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44223 (local address 127.0.0.1:33186)
0927 10:44:37.976822 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:37.976838 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:37.976965 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:37.977945 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:37.977947 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:37.977962 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:37.978066 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:37.978072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.979096 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.979099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:37.980468 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:37.980479 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.981358 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.981361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:37.981461 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:37.981494 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 10:44:37.981629 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 10:44:37.981749 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":22,"spinlock_wait_cycles":5248,"thread_start_us":106,"threads_started":1}
I20260927 10:44:37.982442 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:37.975996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36685 (local address 127.0.0.1:42652)
0927 10:44:37.976766 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:37.976814 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:37.976965 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:37.977887 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:37.977890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:37.977906 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:37.978042 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:37.978050 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.979565 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.979571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:37.980730 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:37.980739 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.981730 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.981734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:37.981822 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:37.981843 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:44:37.981890 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 10:44:37.981966 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":34,"spinlock_wait_cycles":18688,"thread_start_us":78,"threads_started":1}
I20260927 10:44:37.982467 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:37.976158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42652 (local address 127.0.98.1:36685)
0927 10:44:37.977483 (+ 1325us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:37.977500 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:37.977520 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:44:37.977583 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:37.977587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:37.977743 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:37.978185 (+ 442us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:37.978328 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.979445 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.980847 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.981604 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.982064 (+ 460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:37.982192 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:37.982237 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 10:44:37.982316 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1200,"thread_start_us":105,"threads_started":1}
I20260927 10:44:37.983516 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:37.976759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55132 (local address 127.0.98.3:36565)
0927 10:44:37.977248 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:37.977261 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:37.977284 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:44:37.977429 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:37.977435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:37.977628 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:37.977781 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:37.978065 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.979276 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.980698 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.981492 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.981598 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:37.981739 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:37.983299 (+ 1560us) server_negotiation.cc:299] Negotiation successful
0927 10:44:37.983387 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":134,"threads_started":1}
I20260927 10:44:37.983790 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:37.976355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36565 (local address 127.0.0.1:55132)
0927 10:44:37.977200 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:37.977212 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:37.977287 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:37.977787 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:37.977793 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:37.977814 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:37.977950 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:37.977956 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.979400 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.979403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:37.980579 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:37.980587 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:37.981786 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:37.981789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:37.981856 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:37.981885 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 10:44:37.981944 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 10:44:37.981990 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"mutex_wait_us":36,"thread_start_us":99,"threads_started":1}
I20260927 10:44:37.985176 10897 tablet_service.cc:1467] Processing CreateTablet for tablet 055572600457491da74a4b4b00426d62 (DEFAULT_TABLE table=test-table [id=3ba69ea34e4f48aba2b574f710557f0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:37.985175 10763 tablet_service.cc:1467] Processing CreateTablet for tablet 055572600457491da74a4b4b00426d62 (DEFAULT_TABLE table=test-table [id=3ba69ea34e4f48aba2b574f710557f0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:37.986100 10763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 055572600457491da74a4b4b00426d62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:37.986101 10897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 055572600457491da74a4b4b00426d62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:37.991986 11114 tablet_bootstrap.cc:492] T 055572600457491da74a4b4b00426d62 P dd0f89628d57420e99f893bf5140052d: Bootstrap starting.
I20260927 10:44:37.992159 11031 tablet_service.cc:1467] Processing CreateTablet for tablet 055572600457491da74a4b4b00426d62 (DEFAULT_TABLE table=test-table [id=3ba69ea34e4f48aba2b574f710557f0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:37.993165 11031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 055572600457491da74a4b4b00426d62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:37.993186 11115 tablet_bootstrap.cc:492] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0: Bootstrap starting.
I20260927 10:44:37.994269 11114 tablet_bootstrap.cc:654] T 055572600457491da74a4b4b00426d62 P dd0f89628d57420e99f893bf5140052d: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:37.995646 11115 tablet_bootstrap.cc:654] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:37.996153 11114 log.cc:824] T 055572600457491da74a4b4b00426d62 P dd0f89628d57420e99f893bf5140052d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:37.996372 11115 log.cc:824] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:37.998006 11114 tablet_bootstrap.cc:492] T 055572600457491da74a4b4b00426d62 P dd0f89628d57420e99f893bf5140052d: No bootstrap required, opened a new log
I20260927 10:44:37.998222 11114 ts_tablet_manager.cc:1397] T 055572600457491da74a4b4b00426d62 P dd0f89628d57420e99f893bf5140052d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:44:37.998299 11115 tablet_bootstrap.cc:492] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0: No bootstrap required, opened a new log
I20260927 10:44:37.998493 11115 ts_tablet_manager.cc:1397] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:44:37.999605 11118 tablet_bootstrap.cc:492] T 055572600457491da74a4b4b00426d62 P 67f807034d2e43d591e068c05ac674c4: Bootstrap starting.
I20260927 10:44:38.001884 11115 raft_consensus.cc:348] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd0f89628d57420e99f893bf5140052d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36685 } } peers { permanent_uuid: "08c6f86a2dac4810aa8571683b0932b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44223 } } peers { permanent_uuid: "67f807034d2e43d591e068c05ac674c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260927 10:44:38.001824 11114 raft_consensus.cc:348] T 055572600457491da74a4b4b00426d62 P dd0f89628d57420e99f893bf5140052d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd0f89628d57420e99f893bf5140052d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36685 } } peers { permanent_uuid: "08c6f86a2dac4810aa8571683b0932b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44223 } } peers { permanent_uuid: "67f807034d2e43d591e068c05ac674c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260927 10:44:38.002120 11115 raft_consensus.cc:374] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:38.002122 11114 raft_consensus.cc:374] T 055572600457491da74a4b4b00426d62 P dd0f89628d57420e99f893bf5140052d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:38.002163 11118 tablet_bootstrap.cc:654] T 055572600457491da74a4b4b00426d62 P 67f807034d2e43d591e068c05ac674c4: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:38.002177 11115 raft_consensus.cc:729] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08c6f86a2dac4810aa8571683b0932b0, State: Initialized, Role: FOLLOWER
I20260927 10:44:38.002177 11114 raft_consensus.cc:729] T 055572600457491da74a4b4b00426d62 P dd0f89628d57420e99f893bf5140052d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd0f89628d57420e99f893bf5140052d, State: Initialized, Role: FOLLOWER
I20260927 10:44:38.002625 11115 consensus_queue.cc:260] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd0f89628d57420e99f893bf5140052d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36685 } } peers { permanent_uuid: "08c6f86a2dac4810aa8571683b0932b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44223 } } peers { permanent_uuid: "67f807034d2e43d591e068c05ac674c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260927 10:44:38.002699 11114 consensus_queue.cc:260] T 055572600457491da74a4b4b00426d62 P dd0f89628d57420e99f893bf5140052d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd0f89628d57420e99f893bf5140052d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36685 } } peers { permanent_uuid: "08c6f86a2dac4810aa8571683b0932b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44223 } } peers { permanent_uuid: "67f807034d2e43d591e068c05ac674c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260927 10:44:38.003279 10963 heartbeater.cc:503] Master 127.0.98.62:38335 was elected leader, sending a full tablet report...
I20260927 10:44:38.003408 11118 log.cc:824] T 055572600457491da74a4b4b00426d62 P 67f807034d2e43d591e068c05ac674c4: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:38.003496 10829 heartbeater.cc:503] Master 127.0.98.62:38335 was elected leader, sending a full tablet report...
I20260927 10:44:38.003654 11115 ts_tablet_manager.cc:1428] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:44:38.005391 11118 tablet_bootstrap.cc:492] T 055572600457491da74a4b4b00426d62 P 67f807034d2e43d591e068c05ac674c4: No bootstrap required, opened a new log
I20260927 10:44:38.005633 11118 ts_tablet_manager.cc:1397] T 055572600457491da74a4b4b00426d62 P 67f807034d2e43d591e068c05ac674c4: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260927 10:44:38.005779 11114 ts_tablet_manager.cc:1428] T 055572600457491da74a4b4b00426d62 P dd0f89628d57420e99f893bf5140052d: Time spent starting tablet: real 0.007s user 0.000s sys 0.007s
I20260927 10:44:38.009235 11118 raft_consensus.cc:348] T 055572600457491da74a4b4b00426d62 P 67f807034d2e43d591e068c05ac674c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd0f89628d57420e99f893bf5140052d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36685 } } peers { permanent_uuid: "08c6f86a2dac4810aa8571683b0932b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44223 } } peers { permanent_uuid: "67f807034d2e43d591e068c05ac674c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260927 10:44:38.009466 11118 raft_consensus.cc:374] T 055572600457491da74a4b4b00426d62 P 67f807034d2e43d591e068c05ac674c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:38.009500 11118 raft_consensus.cc:729] T 055572600457491da74a4b4b00426d62 P 67f807034d2e43d591e068c05ac674c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67f807034d2e43d591e068c05ac674c4, State: Initialized, Role: FOLLOWER
I20260927 10:44:38.009857 11118 consensus_queue.cc:260] T 055572600457491da74a4b4b00426d62 P 67f807034d2e43d591e068c05ac674c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd0f89628d57420e99f893bf5140052d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36685 } } peers { permanent_uuid: "08c6f86a2dac4810aa8571683b0932b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44223 } } peers { permanent_uuid: "67f807034d2e43d591e068c05ac674c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260927 10:44:38.010463 11097 heartbeater.cc:503] Master 127.0.98.62:38335 was elected leader, sending a full tablet report...
I20260927 10:44:38.010778 11118 ts_tablet_manager.cc:1428] T 055572600457491da74a4b4b00426d62 P 67f807034d2e43d591e068c05ac674c4: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20260927 10:44:38.113001 11119 raft_consensus.cc:482] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:44:38.113248 11119 raft_consensus.cc:504] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd0f89628d57420e99f893bf5140052d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36685 } } peers { permanent_uuid: "08c6f86a2dac4810aa8571683b0932b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44223 } } peers { permanent_uuid: "67f807034d2e43d591e068c05ac674c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260927 10:44:38.114462 11119 leader_election.cc:288] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd0f89628d57420e99f893bf5140052d (127.0.98.1:36685), 67f807034d2e43d591e068c05ac674c4 (127.0.98.3:36565)
I20260927 10:44:38.118472 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:38.114841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48705 (local address 127.0.98.1:36685)
0927 10:44:38.115018 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:38.115022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:38.115039 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:44:38.115216 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:38.115222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:38.115286 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:38.115366 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:38.115726 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:38.116357 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:38.117445 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:38.117850 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:38.117929 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:38.118005 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:38.118234 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 10:44:38.118305 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 10:44:38.118490 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:38.114749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36685 (local address 127.0.98.2:48705)
0927 10:44:38.114976 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:38.114993 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:38.115098 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:38.115431 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:38.115434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:38.115453 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:38.115592 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:38.115598 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:38.116472 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:38.116475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:38.117308 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:38.117316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:38.117984 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:38.117989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:38.118092 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:38.118122 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 10:44:38.118193 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 10:44:38.118306 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260927 10:44:38.118490 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:38.114847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36253 (local address 127.0.98.3:36565)
0927 10:44:38.114978 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:38.114983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:38.115311 (+ 328us) server_negotiation.cc:407] Connection header received
0927 10:44:38.115432 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:38.115435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:38.115485 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:38.115574 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:38.115989 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:38.116490 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:38.117495 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:38.117879 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:38.118030 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:38.118090 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:38.118283 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 10:44:38.118354 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:44:38.118513 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:38.114749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36565 (local address 127.0.98.2:36253)
0927 10:44:38.115182 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:38.115195 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:38.115305 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:38.115721 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:38.115724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:38.115742 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:38.115856 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:38.115863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:38.116606 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:38.116609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:38.117344 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:38.117352 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:38.118061 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:38.118064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:38.118138 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:38.118155 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 10:44:38.118250 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 10:44:38.118311 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":31,"thread_start_us":135,"threads_started":1}
I20260927 10:44:38.119143 10783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055572600457491da74a4b4b00426d62" candidate_uuid: "08c6f86a2dac4810aa8571683b0932b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd0f89628d57420e99f893bf5140052d" is_pre_election: true
I20260927 10:44:38.119355 11051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055572600457491da74a4b4b00426d62" candidate_uuid: "08c6f86a2dac4810aa8571683b0932b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67f807034d2e43d591e068c05ac674c4" is_pre_election: true
I20260927 10:44:38.119436 10783 raft_consensus.cc:2454] T 055572600457491da74a4b4b00426d62 P dd0f89628d57420e99f893bf5140052d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08c6f86a2dac4810aa8571683b0932b0 in term 0.
I20260927 10:44:38.119630 11051 raft_consensus.cc:2454] T 055572600457491da74a4b4b00426d62 P 67f807034d2e43d591e068c05ac674c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08c6f86a2dac4810aa8571683b0932b0 in term 0.
I20260927 10:44:38.120031 10850 leader_election.cc:302] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 08c6f86a2dac4810aa8571683b0932b0, 67f807034d2e43d591e068c05ac674c4, dd0f89628d57420e99f893bf5140052d; no voters:
I20260927 10:44:38.120301 11119 raft_consensus.cc:2781] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:44:38.120374 11119 raft_consensus.cc:482] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:44:38.120426 11119 raft_consensus.cc:3037] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:38.121420 11119 raft_consensus.cc:504] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd0f89628d57420e99f893bf5140052d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36685 } } peers { permanent_uuid: "08c6f86a2dac4810aa8571683b0932b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44223 } } peers { permanent_uuid: "67f807034d2e43d591e068c05ac674c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260927 10:44:38.121762 11119 leader_election.cc:288] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0 [CANDIDATE]: Term 1 election: Requested vote from peers dd0f89628d57420e99f893bf5140052d (127.0.98.1:36685), 67f807034d2e43d591e068c05ac674c4 (127.0.98.3:36565)
I20260927 10:44:38.122104 11051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055572600457491da74a4b4b00426d62" candidate_uuid: "08c6f86a2dac4810aa8571683b0932b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67f807034d2e43d591e068c05ac674c4"
I20260927 10:44:38.122104 10783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055572600457491da74a4b4b00426d62" candidate_uuid: "08c6f86a2dac4810aa8571683b0932b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd0f89628d57420e99f893bf5140052d"
I20260927 10:44:38.122220 11051 raft_consensus.cc:3037] T 055572600457491da74a4b4b00426d62 P 67f807034d2e43d591e068c05ac674c4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:38.122220 10783 raft_consensus.cc:3037] T 055572600457491da74a4b4b00426d62 P dd0f89628d57420e99f893bf5140052d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:38.123461 11051 raft_consensus.cc:2454] T 055572600457491da74a4b4b00426d62 P 67f807034d2e43d591e068c05ac674c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08c6f86a2dac4810aa8571683b0932b0 in term 1.
I20260927 10:44:38.123467 10783 raft_consensus.cc:2454] T 055572600457491da74a4b4b00426d62 P dd0f89628d57420e99f893bf5140052d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08c6f86a2dac4810aa8571683b0932b0 in term 1.
I20260927 10:44:38.123759 10851 leader_election.cc:302] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08c6f86a2dac4810aa8571683b0932b0, 67f807034d2e43d591e068c05ac674c4; no voters:
I20260927 10:44:38.123976 11119 raft_consensus.cc:2781] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:38.124208 11119 raft_consensus.cc:686] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0 [term 1 LEADER]: Becoming Leader. State: Replica: 08c6f86a2dac4810aa8571683b0932b0, State: Running, Role: LEADER
I20260927 10:44:38.124557 11119 consensus_queue.cc:237] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd0f89628d57420e99f893bf5140052d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36685 } } peers { permanent_uuid: "08c6f86a2dac4810aa8571683b0932b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44223 } } peers { permanent_uuid: "67f807034d2e43d591e068c05ac674c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } }
I20260927 10:44:38.127658 10633 catalog_manager.cc:5495] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08c6f86a2dac4810aa8571683b0932b0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "08c6f86a2dac4810aa8571683b0932b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd0f89628d57420e99f893bf5140052d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08c6f86a2dac4810aa8571683b0932b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67f807034d2e43d591e068c05ac674c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 } health_report { overall_health: UNKNOWN } } }
I20260927 10:44:38.174003 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:38.170104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33192 (local address 127.0.98.2:44223)
0927 10:44:38.170251 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:38.170256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:38.170272 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:44:38.170356 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:38.170359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:38.170466 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:38.170613 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:44:38.170852 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:38.171354 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:38.172496 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:38.172726 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:38.172833 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:38.173168 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:44:38.173584 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:44:38.173715 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:38.173778 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 10:44:38.173833 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 10:44:38.174364 11098 tablet.cc:2369] T 055572600457491da74a4b4b00426d62 P 67f807034d2e43d591e068c05ac674c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:44:38.180780 11051 raft_consensus.cc:1261] T 055572600457491da74a4b4b00426d62 P 67f807034d2e43d591e068c05ac674c4 [term 1 FOLLOWER]: Refusing update from remote peer 08c6f86a2dac4810aa8571683b0932b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:38.181408 10783 raft_consensus.cc:1261] T 055572600457491da74a4b4b00426d62 P dd0f89628d57420e99f893bf5140052d [term 1 FOLLOWER]: Refusing update from remote peer 08c6f86a2dac4810aa8571683b0932b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:38.181609 11124 consensus_queue.cc:1035] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67f807034d2e43d591e068c05ac674c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:38.182075 11119 consensus_queue.cc:1035] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd0f89628d57420e99f893bf5140052d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:38.190294 11129 mvcc.cc:205] Tried to move back new op lower bound from 7333912077018886144 to 7333912076800618496. Current Snapshot: MvccSnapshot[applied={T|T < 7333912077018886144}]
I20260927 10:44:38.192087 11128 mvcc.cc:205] Tried to move back new op lower bound from 7333912077018886144 to 7333912076800618496. Current Snapshot: MvccSnapshot[applied={T|T < 7333912077018886144}]
I20260927 10:44:38.198338 10637 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:38.198582 10637 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:38.201048 10637 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2533753efdc94fa4b81d4bc449ef2a4d: Sending DeleteTablet for 3 replicas of tablet 055572600457491da74a4b4b00426d62
I20260927 10:44:38.201879 10763 tablet_service.cc:1514] Processing DeleteTablet for tablet 055572600457491da74a4b4b00426d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42652
I20260927 10:44:38.201874 10897 tablet_service.cc:1514] Processing DeleteTablet for tablet 055572600457491da74a4b4b00426d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33186
I20260927 10:44:38.202055 11031 tablet_service.cc:1514] Processing DeleteTablet for tablet 055572600457491da74a4b4b00426d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55132
I20260927 10:44:38.202615 11137 tablet_replica.cc:330] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0: stopping tablet replica
I20260927 10:44:38.202600 11138 tablet_replica.cc:330] T 055572600457491da74a4b4b00426d62 P dd0f89628d57420e99f893bf5140052d: stopping tablet replica
I20260927 10:44:38.203034 11137 raft_consensus.cc:2229] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:44:38.203126 11138 raft_consensus.cc:2229] T 055572600457491da74a4b4b00426d62 P dd0f89628d57420e99f893bf5140052d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:38.203174 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 10700
I20260927 10:44:38.203464 11138 raft_consensus.cc:2258] T 055572600457491da74a4b4b00426d62 P dd0f89628d57420e99f893bf5140052d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:38.203567 11137 raft_consensus.cc:2258] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:38.205154 11139 tablet_replica.cc:330] T 055572600457491da74a4b4b00426d62 P 67f807034d2e43d591e068c05ac674c4: stopping tablet replica
I20260927 10:44:38.205439 11139 raft_consensus.cc:2229] T 055572600457491da74a4b4b00426d62 P 67f807034d2e43d591e068c05ac674c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:38.205708 11139 raft_consensus.cc:2258] T 055572600457491da74a4b4b00426d62 P 67f807034d2e43d591e068c05ac674c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:38.207186 11139 ts_tablet_manager.cc:1905] T 055572600457491da74a4b4b00426d62 P 67f807034d2e43d591e068c05ac674c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:38.208211 11137 ts_tablet_manager.cc:1905] T 055572600457491da74a4b4b00426d62 P 08c6f86a2dac4810aa8571683b0932b0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 10:44:38.208985 10622 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36685 (error 108)
I20260927 10:44:38.209739 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 10834
I20260927 10:44:38.212997 11139 ts_tablet_manager.cc:1918] T 055572600457491da74a4b4b00426d62 P 67f807034d2e43d591e068c05ac674c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:44:38.213124 11139 log.cc:1196] T 055572600457491da74a4b4b00426d62 P 67f807034d2e43d591e068c05ac674c4: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/055572600457491da74a4b4b00426d62
I20260927 10:44:38.213418 11139 ts_tablet_manager.cc:1939] T 055572600457491da74a4b4b00426d62 P 67f807034d2e43d591e068c05ac674c4: Deleting consensus metadata
I20260927 10:44:38.214628 10623 catalog_manager.cc:4841] TS 67f807034d2e43d591e068c05ac674c4 (127.0.98.3:36565): tablet 055572600457491da74a4b4b00426d62 (table test-table [id=3ba69ea34e4f48aba2b574f710557f0e]) successfully deleted
I20260927 10:44:38.215458 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:38.215150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36685 (local address 127.0.0.1:42658)
0927 10:44:38.215283 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:38.215363 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 10:44:38.215641 10622 catalog_manager.cc:4584] TS dd0f89628d57420e99f893bf5140052d (127.0.98.1:36685): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 055572600457491da74a4b4b00426d62: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36685: connect: Connection refused (error 111)
I20260927 10:44:38.217736 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 10968
I20260927 10:44:38.217916 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:38.217642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44223 (local address 127.0.0.1:33208)
0927 10:44:38.217755 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:38.217837 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 10:44:38.223098 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 10606
2026-09-27T10:44:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2359 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:44:38 dist-test-slave-dzhw krb5kdc[11143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:44:38 dist-test-slave-dzhw krb5kdc[11143](info): set up 2 sockets
Sep 27 10:44:38 dist-test-slave-dzhw krb5kdc[11143](info): commencing operation
krb5kdc: starting...
W20260927 10:44:39.251472 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:44:39 dist-test-slave-dzhw krb5kdc[11143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:44:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:44:39Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 10:44:39.419572 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskkqkM3n/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/postgres -l logfile start
2026-09-27 10:44:42.382 UTC [11173] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 10:44:42.382 UTC [11173] LOG: listening on IPv4 address "127.0.98.20", port 44759
2026-09-27 10:44:42.386 UTC [11173] LOG: listening on Unix socket "/tmp/.s.PGSQL.44759"
2026-09-27 10:44:42.421 UTC [11176] LOG: database system was shut down at 2026-09-27 10:44:40 UTC
2026-09-27 10:44:42.425 UTC [11173] LOG: database system is ready to accept connections
I20260927 10:44:43.363125 392 mini_postgres.cc:96] Postgres bound to 44759
2026-09-27 10:44:43.369 UTC [11184] FATAL: database "slave" does not exist
127.0.98.20:44759 - accepting connections
I20260927 10:44:43.370218 392 mini_ranger.cc:161] Starting Ranger...
I20260927 10:44:43.385700 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 10:44:43.885353 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 10:44:43.885437 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ranger-admin
2026-09-27 10:44:44,161 [I] DB FLAVOR :POSTGRES
2026-09-27 10:44:44,161 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 10:44:44,161 [I] Checking connection..
2026-09-27 10:44:44,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 10:44:44,408 [I] Checking connection passed.
2026-09-27 10:44:44,409 [I] --------- Verifying version history table ---------
2026-09-27 10:44:44,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:44:44,658 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 10:44:44,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:44:44,923 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 10:44:44,923 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 10:44:44,923 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44759/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 10:44:45.146 UTC [11248] WARNING: there is no transaction in progress
2026-09-27 10:44:45,153 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 10:44:45,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:44:45,385 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 10:44:45,385 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 10:44:45,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 10:44:45,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 10:44:45,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dzhw.c.gcp-upstream.internal','N') ;"
2026-09-27 10:44:46,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 10:44:46,370 [I] Table x_portal_user does not exist in database ranger
2026-09-27 10:44:46,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 10:44:46,601 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 10:44:46,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 10:44:46,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 10:44:47,087 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 10:44:47,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44759/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 10:44:48.069 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 10:44:48.084 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 10:44:48.100 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 10:44:48.118 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 10:44:48.133 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 10:44:48.152 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 10:44:48.209 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 10:44:48.216 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 10:44:48.230 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 10:44:48.241 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 10:44:48.251 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 10:44:48.267 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 10:44:48.278 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 10:44:48.285 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 10:44:48.291 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 10:44:48.298 UTC [11356] WARNING: there is no transaction in progress
2026-09-27 10:44:48,992 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 10:44:48,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 10:44:49,221 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 10:44:49,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 10:44:49,458 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 10:44:49,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 10:44:49,689 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 10:44:49,689 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 10:44:49,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 10:44:50,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dzhw.c.gcp-upstream.internal';"
2026-09-27 10:44:50,435 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 10:44:50,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44759/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 10:44:50,682 [I] DB_PATCHES have already been applied
I20260927 10:44:50.687605 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskkqkM3n/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 10:44:50.687677 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 10:44:50.691110 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:51171
Sep 27, 2026 10:44:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:44:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:44:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 10:44:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 10:44:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 10:44:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 10:44:51 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 10:44:51.800244 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.430s user 0.002s sys 0.008s
Sep 27, 2026 10:45:11 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 10:45:11 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 10:45:11 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 10:45:11 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 10:45:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 10:45:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 10:45:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 10:45:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 10:45:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 10:45:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 10:45:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 10:45:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 10:45:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 10:45:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 10:45:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 10:45:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 10:45:11 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 10:45:12 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 10:45:12.709900 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 10:45:12 dist-test-slave-dzhw krb5kdc[11143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505912, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 10:45:12.799875 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 10:45:12.811156 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 10:45:13.733317 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 10:45:13.733420 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ranger-kms
I20260927 10:45:13.733521 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:44759
2026-09-27 10:45:13,924 [I] DB FLAVOR :POSTGRES
2026-09-27 10:45:13,924 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 10:45:13,924 [I] Checking connection
2026-09-27 10:45:13,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44759/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 10:45:14,163 [I] connection success
2026-09-27 10:45:14,163 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 10:45:14,163 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 10:45:14,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44759/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 10:45:14,409 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 10:45:14,410 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 10:45:14,410 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 10:45:14,410 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44759/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 10:45:14,679 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 10:45:15.164487 392 mini_ranger_kms.cc:310] Created kms service
I20260927 10:45:15.349079 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 10:45:15.396351 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 10:45:15 dist-test-slave-dzhw krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505912, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 10:45:15.795622 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 10:45:15.795759 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 10:45:15.795782 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 10:45:15.799635 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11536
I20260927 10:45:15.799769 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:54339
Sep 27, 2026 10:45:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:45:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:45:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 10:45:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 10:45:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 10:45:17 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 10:45:18.086565 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.287s user 0.010s sys 0.165s
I20260927 10:45:18.086709 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 10:45:18.086824 392 mini_ranger_kms.cc:401] 127.0.98.20:54339/kms/v1/keys
Sep 27 10:45:18 dist-test-slave-dzhw krb5kdc[11143](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790505918, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 10:45:18 dist-test-slave-dzhw krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505918, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 10:45:19 dist-test-slave-dzhw krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505918, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 10:45:21 dist-test-slave-dzhw krb5kdc[11143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:45:21.278481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33567
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33567
--ranger_config_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54339/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:21.415014 11587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:21.415324 11587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:21.415367 11587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:21.420807 11587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:45:21.420895 11587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:21.420917 11587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:21.420933 11587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:45:21.420948 11587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:45:21.427083 11587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:54339/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33567
--ranger_config_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:45:21.428264 11587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:21.429349 11587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:21.439915 11592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:21.439915 11595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:21.440168 11593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:21.441159 11587 server_base.cc:1034] running on GCE node
I20260927 10:45:21.441883 11587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:21.443550 11587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:21.444742 11587 hybrid_clock.cc:648] HybridClock initialized: now 1790505921444703 us; error 45 us; skew 500 ppm
Sep 27 10:45:21 dist-test-slave-dzhw krb5kdc[11143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:21.449229 11587 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:45:21.450456 11587 webserver.cc:460] Webserver started at http://127.0.98.62:33029/ using document root <none> and password file <none>
I20260927 10:45:21.451211 11587 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:21.451308 11587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:21.451583 11587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:45:21 dist-test-slave-dzhw krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 10:45:21.514635 11587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "5200fb6a5f124d09aaba410f254001cb"
format_stamp: "Formatted at 2026-09-27 10:45:21 on dist-test-slave-dzhw"
server_key: "ea3af0348b0964063a4ce5bfc0fc9785"
server_key_iv: "4ba9004319cf681c63ebf1eff169a5ea"
server_key_version: "kuduclusterkey@0"
I20260927 10:45:21.515363 11587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "5200fb6a5f124d09aaba410f254001cb"
format_stamp: "Formatted at 2026-09-27 10:45:21 on dist-test-slave-dzhw"
server_key: "ea3af0348b0964063a4ce5bfc0fc9785"
server_key_iv: "4ba9004319cf681c63ebf1eff169a5ea"
server_key_version: "kuduclusterkey@0"
I20260927 10:45:21.520207 11587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 10:45:21.538539 11602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:21.539974 11587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:45:21.540115 11587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "5200fb6a5f124d09aaba410f254001cb"
format_stamp: "Formatted at 2026-09-27 10:45:21 on dist-test-slave-dzhw"
server_key: "ea3af0348b0964063a4ce5bfc0fc9785"
server_key_iv: "4ba9004319cf681c63ebf1eff169a5ea"
server_key_version: "kuduclusterkey@0"
I20260927 10:45:21.540251 11587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:21.554246 11587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:21.557184 11587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:21.557404 11587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:21.566114 11587 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33567
I20260927 10:45:21.566131 11654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33567 every 8 connection(s)
I20260927 10:45:21.567202 11587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:45:21.567728 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 11587
I20260927 10:45:21.567853 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:45:21.568102 392 external_mini_cluster.cc:1428] Setting key c010da1ea1234e2c1066cf95ead6bdaf
I20260927 10:45:21.570758 11655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 10:45:21 dist-test-slave-dzhw krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:45:21.577940 11655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5200fb6a5f124d09aaba410f254001cb: Bootstrap starting.
I20260927 10:45:21.580740 11655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5200fb6a5f124d09aaba410f254001cb: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:21.582074 11655 log.cc:824] T 00000000000000000000000000000000 P 5200fb6a5f124d09aaba410f254001cb: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:21.582574 11658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.569437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36742 (local address 127.0.98.62:33567)
0927 10:45:21.569860 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:21.569881 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:21.570142 (+ 261us) server_negotiation.cc:407] Connection header received
0927 10:45:21.571067 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:21.571085 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:21.571413 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:21.571769 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:21.572559 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:21.573549 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:21.574406 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:21.574911 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:21.575111 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:21.577753 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:21.577758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:21.577773 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:21.577800 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:21.579535 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:21.579997 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:21.580003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:21.580016 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:21.580119 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:21.580411 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:21.580414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:21.580416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:21.580888 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:21.581095 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:21.581403 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 10:45:21.581700 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":147,"threads_started":1}
I20260927 10:45:21.584247 11655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5200fb6a5f124d09aaba410f254001cb: No bootstrap required, opened a new log
I20260927 10:45:21.587471 11655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5200fb6a5f124d09aaba410f254001cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5200fb6a5f124d09aaba410f254001cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33567 } }
I20260927 10:45:21.587760 11655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5200fb6a5f124d09aaba410f254001cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:21.587845 11655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5200fb6a5f124d09aaba410f254001cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5200fb6a5f124d09aaba410f254001cb, State: Initialized, Role: FOLLOWER
I20260927 10:45:21.588327 11655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5200fb6a5f124d09aaba410f254001cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5200fb6a5f124d09aaba410f254001cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33567 } }
I20260927 10:45:21.588485 11655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5200fb6a5f124d09aaba410f254001cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:45:21.588568 11655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5200fb6a5f124d09aaba410f254001cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:45:21.588668 11655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5200fb6a5f124d09aaba410f254001cb [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:21.589747 11655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5200fb6a5f124d09aaba410f254001cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5200fb6a5f124d09aaba410f254001cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33567 } }
I20260927 10:45:21.590128 11655 leader_election.cc:302] T 00000000000000000000000000000000 P 5200fb6a5f124d09aaba410f254001cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5200fb6a5f124d09aaba410f254001cb; no voters:
I20260927 10:45:21.590538 11655 leader_election.cc:288] T 00000000000000000000000000000000 P 5200fb6a5f124d09aaba410f254001cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:45:21.590638 11660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5200fb6a5f124d09aaba410f254001cb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:21.590874 11660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5200fb6a5f124d09aaba410f254001cb [term 1 LEADER]: Becoming Leader. State: Replica: 5200fb6a5f124d09aaba410f254001cb, State: Running, Role: LEADER
I20260927 10:45:21.591315 11660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5200fb6a5f124d09aaba410f254001cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5200fb6a5f124d09aaba410f254001cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33567 } }
I20260927 10:45:21.591902 11655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5200fb6a5f124d09aaba410f254001cb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:45:21.593248 11661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5200fb6a5f124d09aaba410f254001cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5200fb6a5f124d09aaba410f254001cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5200fb6a5f124d09aaba410f254001cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33567 } } }
I20260927 10:45:21.593374 11661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5200fb6a5f124d09aaba410f254001cb [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:21.593601 11661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5200fb6a5f124d09aaba410f254001cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5200fb6a5f124d09aaba410f254001cb. Latest consensus state: current_term: 1 leader_uuid: "5200fb6a5f124d09aaba410f254001cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5200fb6a5f124d09aaba410f254001cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33567 } } }
I20260927 10:45:21.593675 11661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5200fb6a5f124d09aaba410f254001cb [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:21.594010 11671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:45:21.594182 11655 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 10:45:21.603551 11671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:45:21.612735 11671 catalog_manager.cc:1347] Generated new cluster ID: d6698d94b3d5410e9cd99619b146f1dd
I20260927 10:45:21.612805 11671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:45:21.624800 11671 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:45:21.626211 11671 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:45:21.632098 11671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5200fb6a5f124d09aaba410f254001cb: Generated new TSK 0
I20260927 10:45:21.633009 11671 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 10:45:22.397410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54339/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33567
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:22.551270 11700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:22.551529 11700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:22.551573 11700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:22.556929 11700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:22.557022 11700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:22.557111 11700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:45:22.564579 11700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:54339/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:45:22.565869 11700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:22.567193 11700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:22.580716 11707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:22.580717 11710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:22.580912 11708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:22.581463 11700 server_base.cc:1034] running on GCE node
I20260927 10:45:22.581889 11700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:22.582623 11700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:22.583801 11700 hybrid_clock.cc:648] HybridClock initialized: now 1790505922583769 us; error 52 us; skew 500 ppm
Sep 27 10:45:22 dist-test-slave-dzhw krb5kdc[11143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:22.586710 11700 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 10:45:22.587741 11700 webserver.cc:460] Webserver started at http://127.0.98.1:35479/ using document root <none> and password file <none>
I20260927 10:45:22.588320 11700 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:22.588410 11700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:22.588651 11700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:45:22 dist-test-slave-dzhw krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 10:45:22.605871 11700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "a8ca3a6958f44730b3951170de3fbbef"
format_stamp: "Formatted at 2026-09-27 10:45:22 on dist-test-slave-dzhw"
server_key: "53078ec86caaecadda099db59d798013"
server_key_iv: "a0b951cddcef2712a41ccef54e7835d9"
server_key_version: "kuduclusterkey@0"
I20260927 10:45:22.606832 11700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a8ca3a6958f44730b3951170de3fbbef"
format_stamp: "Formatted at 2026-09-27 10:45:22 on dist-test-slave-dzhw"
server_key: "53078ec86caaecadda099db59d798013"
server_key_iv: "a0b951cddcef2712a41ccef54e7835d9"
server_key_version: "kuduclusterkey@0"
I20260927 10:45:22.613176 11700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 10:45:22.635627 11717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:22.637396 11700 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260927 10:45:22.637537 11700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "a8ca3a6958f44730b3951170de3fbbef"
format_stamp: "Formatted at 2026-09-27 10:45:22 on dist-test-slave-dzhw"
server_key: "53078ec86caaecadda099db59d798013"
server_key_iv: "a0b951cddcef2712a41ccef54e7835d9"
server_key_version: "kuduclusterkey@0"
I20260927 10:45:22.637686 11700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:22.652853 11700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:22.656510 11700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:22.656785 11700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:22.657523 11700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:22.658648 11700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:22.658761 11700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:22.658886 11700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:22.658968 11700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:22.669684 11700 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44079
I20260927 10:45:22.670899 11700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:45:22.677201 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 11700
I20260927 10:45:22.677325 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:45:22.677616 392 external_mini_cluster.cc:1428] Setting key 792da4e24680c687f023b79fb753aa39
I20260927 10:45:22.681720 11832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44079 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 27 10:45:22 dist-test-slave-dzhw krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 10:45:22.712075 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:22.684991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33567 (local address 127.0.98.1:54433)
0927 10:45:22.692267 (+ 7276us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:22.692347 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:22.693492 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:22.694548 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:22.694558 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:22.694933 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:22.695624 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:22.695642 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:22.697409 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:22.697413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:22.698867 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:22.698878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:22.699714 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:22.699718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:22.699918 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:22.700629 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:22.700650 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:22.706115 (+ 5465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:22.708609 (+ 2494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:22.708617 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:22.708636 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:22.709051 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:22.709396 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:22.709399 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:22.709401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:22.709596 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:22.710087 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:22.710094 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:22.710478 (+ 384us) client_negotiation.cc:769] Sending connection context
0927 10:45:22.710682 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 10:45:22.711028 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6984,"thread_start_us":141,"threads_started":1}
I20260927 10:45:22.712814 11836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:22.684485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54433 (local address 127.0.98.62:33567)
0927 10:45:22.693995 (+ 9510us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:22.694000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:22.694016 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:45:22.694061 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:22.694064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:22.694115 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:22.694203 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:22.695804 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:22.696689 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:22.699049 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:22.699412 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:22.699516 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:22.706363 (+ 6847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:22.706367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:22.706370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:22.706400 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:22.708450 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:22.709193 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:22.709197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:22.709199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:22.709292 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:22.709712 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:22.709715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:22.709717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:22.709948 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:22.710035 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:22.712532 (+ 2497us) server_negotiation.cc:299] Negotiation successful
0927 10:45:22.712662 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9413,"thread_start_us":109,"threads_started":1}
I20260927 10:45:22.714622 11833 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33567
I20260927 10:45:22.714864 11833 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:22.715428 11833 heartbeater.cc:511] Master 127.0.98.62:33567 requested a full tablet report, sending...
I20260927 10:45:22.717268 11619 ts_manager.cc:194] Registered new tserver with Master: a8ca3a6958f44730b3951170de3fbbef (127.0.98.1:44079)
I20260927 10:45:22.718787 11619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 10:45:22.737421 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54339/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33567
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:45:22 dist-test-slave-dzhw krb5kdc[11143](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790505922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 10:45:22.901762 11839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:22.902184 11839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:22.902258 11839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:22.908154 11839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:22.908239 11839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:22.908325 11839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:45:22.916138 11839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:54339/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:45:22.917541 11839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:22.918839 11839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:22.931274 11848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:22.931267 11846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:22.931404 11845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:22.931457 11839 server_base.cc:1034] running on GCE node
I20260927 10:45:22.932176 11839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:22.932926 11839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:22.934114 11839 hybrid_clock.cc:648] HybridClock initialized: now 1790505922934095 us; error 34 us; skew 500 ppm
Sep 27 10:45:22 dist-test-slave-dzhw krb5kdc[11143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:22.937954 11839 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 10:45:22.939075 11839 webserver.cc:460] Webserver started at http://127.0.98.2:41813/ using document root <none> and password file <none>
I20260927 10:45:22.939750 11839 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:22.939816 11839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:22.940157 11839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:45:22 dist-test-slave-dzhw krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 10:45:22.968773 11839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "01c3c1aefa494b6cb07d9cff54ea0239"
format_stamp: "Formatted at 2026-09-27 10:45:22 on dist-test-slave-dzhw"
server_key: "03b799c4673e717516e392be1487c9f7"
server_key_iv: "f2e680b4c73aac8753ba8c19fd9cd3e1"
server_key_version: "kuduclusterkey@0"
I20260927 10:45:22.969642 11839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "01c3c1aefa494b6cb07d9cff54ea0239"
format_stamp: "Formatted at 2026-09-27 10:45:22 on dist-test-slave-dzhw"
server_key: "03b799c4673e717516e392be1487c9f7"
server_key_iv: "f2e680b4c73aac8753ba8c19fd9cd3e1"
server_key_version: "kuduclusterkey@0"
I20260927 10:45:22.977778 11839 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260927 10:45:22.998848 11855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:23.000846 11839 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260927 10:45:23.001082 11839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "01c3c1aefa494b6cb07d9cff54ea0239"
format_stamp: "Formatted at 2026-09-27 10:45:22 on dist-test-slave-dzhw"
server_key: "03b799c4673e717516e392be1487c9f7"
server_key_iv: "f2e680b4c73aac8753ba8c19fd9cd3e1"
server_key_version: "kuduclusterkey@0"
I20260927 10:45:23.001322 11839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:23.020412 11839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:23.024863 11839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:23.025135 11839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:23.025913 11839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:23.027083 11839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:23.027191 11839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:23.027302 11839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:23.027374 11839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:23.041899 11839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40449
I20260927 10:45:23.043295 11839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:45:23.045293 11968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40449 every 8 connection(s)
I20260927 10:45:23.050980 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 11839
I20260927 10:45:23.051128 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:45:23.051445 392 external_mini_cluster.cc:1428] Setting key 299db3ee4d145b5f3cc9b8943eade3dd
Sep 27 10:45:23 dist-test-slave-dzhw krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:45:23.066182 11836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.048887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33907 (local address 127.0.98.62:33567)
0927 10:45:23.049045 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:23.049051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:23.049978 (+ 927us) server_negotiation.cc:407] Connection header received
0927 10:45:23.051384 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:23.051389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:23.051449 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:23.051510 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:23.053242 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.054039 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.055913 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.056280 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.056360 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.059567 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:23.059571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:23.059574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:23.059606 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:23.061815 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:23.062536 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:23.062540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:23.062542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:23.062637 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:23.063054 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:23.063057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:23.063061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:23.063317 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:23.063408 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:23.065800 (+ 2392us) server_negotiation.cc:299] Negotiation successful
0927 10:45:23.065959 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:45:23.068084 11971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.047686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33567 (local address 127.0.98.2:33907)
0927 10:45:23.049805 (+ 2119us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:23.049863 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:23.050985 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:23.051972 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:23.051982 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:23.052355 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:23.053059 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.053091 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.054238 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.054244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.055762 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.055773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.056568 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.056573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.056778 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.057531 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:23.057553 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:23.059352 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:23.061964 (+ 2612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:23.061970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:23.061989 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:23.062367 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:23.062747 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:23.062751 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:23.062753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:23.062936 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:23.063476 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:23.063482 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:23.063837 (+ 355us) client_negotiation.cc:769] Sending connection context
0927 10:45:23.066534 (+ 2697us) client_negotiation.cc:240] Negotiation successful
0927 10:45:23.066965 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1892,"thread_start_us":156,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 10:45:23.073630 11969 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33567
I20260927 10:45:23.073907 11969 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:23.074620 11969 heartbeater.cc:511] Master 127.0.98.62:33567 requested a full tablet report, sending...
I20260927 10:45:23.075942 11619 ts_manager.cc:194] Registered new tserver with Master: 01c3c1aefa494b6cb07d9cff54ea0239 (127.0.98.2:40449)
I20260927 10:45:23.076738 11619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 10:45:23.103622 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54339/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33567
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:23.267660 11974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:23.268111 11974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:23.268224 11974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:23.274185 11974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:23.274335 11974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:23.274505 11974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:45:23.282227 11974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:54339/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:45:23.283840 11974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:23.285185 11974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:23.299494 11979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:23.299875 11980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:23.302870 11982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:23.313261 11974 server_base.cc:1034] running on GCE node
I20260927 10:45:23.313830 11974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:23.314682 11974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:23.315845 11974 hybrid_clock.cc:648] HybridClock initialized: now 1790505923315777 us; error 82 us; skew 500 ppm
Sep 27 10:45:23 dist-test-slave-dzhw krb5kdc[11143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:23.319674 11974 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 10:45:23.320894 11974 webserver.cc:460] Webserver started at http://127.0.98.3:45443/ using document root <none> and password file <none>
I20260927 10:45:23.321687 11974 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:23.321833 11974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:23.322114 11974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:45:23 dist-test-slave-dzhw krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 10:45:23 dist-test-slave-dzhw krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 10:45:23.349303 11974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "c1817f7b66724d389d2ab38ab62a9be2"
format_stamp: "Formatted at 2026-09-27 10:45:23 on dist-test-slave-dzhw"
server_key: "1bcc0ee132deb27e0ae8c1c39cc512da"
server_key_iv: "580bf4e943d762809fa93a57e8b15028"
server_key_version: "kuduclusterkey@0"
I20260927 10:45:23.354408 11974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c1817f7b66724d389d2ab38ab62a9be2"
format_stamp: "Formatted at 2026-09-27 10:45:23 on dist-test-slave-dzhw"
server_key: "1bcc0ee132deb27e0ae8c1c39cc512da"
server_key_iv: "580bf4e943d762809fa93a57e8b15028"
server_key_version: "kuduclusterkey@0"
I20260927 10:45:23.360337 11974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 10:45:23.383631 11989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:23.385331 11974 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 10:45:23.385461 11974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "c1817f7b66724d389d2ab38ab62a9be2"
format_stamp: "Formatted at 2026-09-27 10:45:23 on dist-test-slave-dzhw"
server_key: "1bcc0ee132deb27e0ae8c1c39cc512da"
server_key_iv: "580bf4e943d762809fa93a57e8b15028"
server_key_version: "kuduclusterkey@0"
I20260927 10:45:23.385592 11974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:23.401619 11974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:23.405454 11974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:23.405707 11974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:23.406451 11974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:23.407575 11974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:23.407673 11974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:23.407763 11974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:23.407825 11974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:23.420914 11974 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45875
I20260927 10:45:23.422111 11974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:45:23.423053 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 11974
I20260927 10:45:23.423197 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:45:23.423502 392 external_mini_cluster.cc:1428] Setting key 31e624cb18f4985420c2ebe9b6ef38f0
I20260927 10:45:23.422705 12102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45875 every 8 connection(s)
Sep 27 10:45:23 dist-test-slave-dzhw krb5kdc[11143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:45:23.443776 11836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.428898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44305 (local address 127.0.98.62:33567)
0927 10:45:23.429018 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:23.429022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:23.429504 (+ 482us) server_negotiation.cc:407] Connection header received
0927 10:45:23.430978 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:23.430982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:23.431033 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:23.431081 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:23.432663 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.433478 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.435093 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.435423 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.435493 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.438509 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:23.438513 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:23.438516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:23.438543 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:23.440750 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:23.442000 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:23.442004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:23.442006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:23.442086 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:23.442496 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:23.442500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:23.442502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:23.442712 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:23.442786 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:23.443498 (+ 712us) server_negotiation.cc:299] Negotiation successful
0927 10:45:23.443618 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:45:23.445796 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.427356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33567 (local address 127.0.98.3:44305)
0927 10:45:23.429355 (+ 1999us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:23.429411 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:23.430506 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:23.431480 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:23.431489 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:23.431850 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:23.432499 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.432516 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.433629 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.433632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.434963 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.434974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.435698 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.435701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.435888 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.436571 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:23.436591 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:23.438287 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:23.441478 (+ 3191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:23.441484 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:23.441503 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:23.441890 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:23.442191 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:23.442194 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:23.442196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:23.442367 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:23.442848 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:23.442855 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:23.443202 (+ 347us) client_negotiation.cc:769] Sending connection context
0927 10:45:23.444261 (+ 1059us) client_negotiation.cc:240] Negotiation successful
0927 10:45:23.444657 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1769,"thread_start_us":557,"threads_started":1}
I20260927 10:45:23.447310 12103 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33567
I20260927 10:45:23.447540 12103 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:23.448082 12103 heartbeater.cc:511] Master 127.0.98.62:33567 requested a full tablet report, sending...
I20260927 10:45:23.449214 11619 ts_manager.cc:194] Registered new tserver with Master: c1817f7b66724d389d2ab38ab62a9be2 (127.0.98.3:45875)
I20260927 10:45:23.449926 11619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44305
I20260927 10:45:23.453482 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:45:23.465387 11836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.455738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36758 (local address 127.0.98.62:33567)
0927 10:45:23.455855 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:23.455860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:23.456446 (+ 586us) server_negotiation.cc:407] Connection header received
0927 10:45:23.456622 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:23.456625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:23.456673 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:23.456723 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:23.457682 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.458472 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.459621 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.459936 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.460010 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.461815 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:23.461819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:23.461822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:23.461848 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:23.463566 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:23.464051 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:23.464054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:23.464056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:23.464134 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:23.464429 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:23.464432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:23.464434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:23.464639 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:23.464737 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:23.465080 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 10:45:23.465215 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:45:23.469276 11619 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:45:23.473757 11619 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:45:23.509889 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.488828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44079 (local address 127.0.0.1:58504)
0927 10:45:23.492451 (+ 3623us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:23.492464 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:23.492551 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:23.494188 (+ 1637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:23.494194 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:23.494220 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:23.494374 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.494381 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.499008 (+ 4627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.499012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.500369 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.500380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.509234 (+ 8854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.509239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.509387 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.509433 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 10:45:23.509561 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 10:45:23.509718 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3566,"thread_start_us":85,"threads_started":1}
I20260927 10:45:23.513458 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.489857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45875 (local address 127.0.0.1:60910)
0927 10:45:23.494603 (+ 4746us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:23.494617 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:23.494717 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:23.496591 (+ 1874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:23.497386 (+ 795us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:23.497406 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:23.497492 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.497499 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.506372 (+ 8873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.506376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.510239 (+ 3863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.510250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.513001 (+ 2751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.513004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.513101 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.513126 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:45:23.513215 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 10:45:23.513296 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4662,"thread_start_us":94,"threads_started":1}
I20260927 10:45:23.514878 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.486211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40449 (local address 127.0.0.1:34006)
0927 10:45:23.491829 (+ 5618us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:23.491893 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:23.492028 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:23.496327 (+ 4299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:23.496330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:23.496348 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:23.496447 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.496453 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.503844 (+ 7391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.503848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.505079 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.505091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.514490 (+ 9399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.514494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.514590 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.514611 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:45:23.514674 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 10:45:23.514743 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5333,"thread_start_us":123,"threads_started":1}
I20260927 10:45:23.516232 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.489103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58504 (local address 127.0.98.1:44079)
0927 10:45:23.492320 (+ 3217us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:23.492328 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:23.492838 (+ 510us) server_negotiation.cc:407] Connection header received
0927 10:45:23.492922 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:23.492927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:23.493169 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:23.493319 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:23.497735 (+ 4416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.498867 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.507858 (+ 8991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.508859 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.508981 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.509161 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:23.515933 (+ 6772us) server_negotiation.cc:299] Negotiation successful
0927 10:45:23.516068 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3110,"thread_start_us":134,"threads_started":1}
I20260927 10:45:23.516984 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.487310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34006 (local address 127.0.98.2:40449)
0927 10:45:23.495723 (+ 8413us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:23.495735 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:23.495757 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:45:23.495829 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:23.495834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:23.496015 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:23.496178 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:23.502514 (+ 6336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.503682 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.510674 (+ 6992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.514260 (+ 3586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.514374 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.516683 (+ 2309us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:23.516744 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 10:45:23.516844 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8301,"thread_start_us":107,"threads_started":1}
I20260927 10:45:23.517508 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.490257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60910 (local address 127.0.98.3:45875)
0927 10:45:23.492785 (+ 2528us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:23.492795 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:23.494838 (+ 2043us) server_negotiation.cc:407] Connection header received
0927 10:45:23.494950 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:23.494955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:23.495254 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:23.495413 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:23.501320 (+ 5907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.506228 (+ 4908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.510488 (+ 4260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.512661 (+ 2173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.512778 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.512934 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:23.517245 (+ 4311us) server_negotiation.cc:299] Negotiation successful
0927 10:45:23.517363 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2432,"thread_start_us":84,"threads_started":1}
I20260927 10:45:23.521373 11767 tablet_service.cc:1467] Processing CreateTablet for tablet 702190ed4e0242719e0fa48ebca06892 (DEFAULT_TABLE table=test-table [id=fe4abe98fc3a401c967ff7a0f2ea52f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:23.523789 11767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 702190ed4e0242719e0fa48ebca06892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:23.534950 12036 tablet_service.cc:1467] Processing CreateTablet for tablet 702190ed4e0242719e0fa48ebca06892 (DEFAULT_TABLE table=test-table [id=fe4abe98fc3a401c967ff7a0f2ea52f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:23.536185 12036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 702190ed4e0242719e0fa48ebca06892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:23.546918 11902 tablet_service.cc:1467] Processing CreateTablet for tablet 702190ed4e0242719e0fa48ebca06892 (DEFAULT_TABLE table=test-table [id=fe4abe98fc3a401c967ff7a0f2ea52f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:23.548105 11902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 702190ed4e0242719e0fa48ebca06892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:23.564579 12123 tablet_bootstrap.cc:492] T 702190ed4e0242719e0fa48ebca06892 P c1817f7b66724d389d2ab38ab62a9be2: Bootstrap starting.
I20260927 10:45:23.565068 12122 tablet_bootstrap.cc:492] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239: Bootstrap starting.
I20260927 10:45:23.571066 12124 tablet_bootstrap.cc:492] T 702190ed4e0242719e0fa48ebca06892 P a8ca3a6958f44730b3951170de3fbbef: Bootstrap starting.
I20260927 10:45:23.573858 12122 tablet_bootstrap.cc:654] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:23.576795 12124 tablet_bootstrap.cc:654] T 702190ed4e0242719e0fa48ebca06892 P a8ca3a6958f44730b3951170de3fbbef: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:23.577775 12124 log.cc:824] T 702190ed4e0242719e0fa48ebca06892 P a8ca3a6958f44730b3951170de3fbbef: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:23.578068 12123 tablet_bootstrap.cc:654] T 702190ed4e0242719e0fa48ebca06892 P c1817f7b66724d389d2ab38ab62a9be2: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:23.579727 12122 log.cc:824] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:23.580201 12124 tablet_bootstrap.cc:492] T 702190ed4e0242719e0fa48ebca06892 P a8ca3a6958f44730b3951170de3fbbef: No bootstrap required, opened a new log
I20260927 10:45:23.580410 12124 ts_tablet_manager.cc:1397] T 702190ed4e0242719e0fa48ebca06892 P a8ca3a6958f44730b3951170de3fbbef: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 10:45:23.581434 12123 log.cc:824] T 702190ed4e0242719e0fa48ebca06892 P c1817f7b66724d389d2ab38ab62a9be2: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:23.583690 12122 tablet_bootstrap.cc:492] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239: No bootstrap required, opened a new log
I20260927 10:45:23.583901 12122 ts_tablet_manager.cc:1397] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20260927 10:45:23.584894 12124 raft_consensus.cc:348] T 702190ed4e0242719e0fa48ebca06892 P a8ca3a6958f44730b3951170de3fbbef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c3c1aefa494b6cb07d9cff54ea0239" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40449 } } peers { permanent_uuid: "a8ca3a6958f44730b3951170de3fbbef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44079 } } peers { permanent_uuid: "c1817f7b66724d389d2ab38ab62a9be2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45875 } }
I20260927 10:45:23.585179 12124 raft_consensus.cc:374] T 702190ed4e0242719e0fa48ebca06892 P a8ca3a6958f44730b3951170de3fbbef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:23.585238 12124 raft_consensus.cc:729] T 702190ed4e0242719e0fa48ebca06892 P a8ca3a6958f44730b3951170de3fbbef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8ca3a6958f44730b3951170de3fbbef, State: Initialized, Role: FOLLOWER
I20260927 10:45:23.585762 12124 consensus_queue.cc:260] T 702190ed4e0242719e0fa48ebca06892 P a8ca3a6958f44730b3951170de3fbbef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c3c1aefa494b6cb07d9cff54ea0239" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40449 } } peers { permanent_uuid: "a8ca3a6958f44730b3951170de3fbbef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44079 } } peers { permanent_uuid: "c1817f7b66724d389d2ab38ab62a9be2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45875 } }
I20260927 10:45:23.586913 12124 ts_tablet_manager.cc:1428] T 702190ed4e0242719e0fa48ebca06892 P a8ca3a6958f44730b3951170de3fbbef: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:45:23.587347 11833 heartbeater.cc:503] Master 127.0.98.62:33567 was elected leader, sending a full tablet report...
I20260927 10:45:23.588089 12122 raft_consensus.cc:348] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c3c1aefa494b6cb07d9cff54ea0239" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40449 } } peers { permanent_uuid: "a8ca3a6958f44730b3951170de3fbbef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44079 } } peers { permanent_uuid: "c1817f7b66724d389d2ab38ab62a9be2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45875 } }
I20260927 10:45:23.588356 12122 raft_consensus.cc:374] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:23.588406 12122 raft_consensus.cc:729] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c3c1aefa494b6cb07d9cff54ea0239, State: Initialized, Role: FOLLOWER
I20260927 10:45:23.588876 12122 consensus_queue.cc:260] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c3c1aefa494b6cb07d9cff54ea0239" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40449 } } peers { permanent_uuid: "a8ca3a6958f44730b3951170de3fbbef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44079 } } peers { permanent_uuid: "c1817f7b66724d389d2ab38ab62a9be2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45875 } }
I20260927 10:45:23.589934 12122 ts_tablet_manager.cc:1428] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:45:23.590309 11969 heartbeater.cc:503] Master 127.0.98.62:33567 was elected leader, sending a full tablet report...
I20260927 10:45:23.592299 12123 tablet_bootstrap.cc:492] T 702190ed4e0242719e0fa48ebca06892 P c1817f7b66724d389d2ab38ab62a9be2: No bootstrap required, opened a new log
I20260927 10:45:23.592523 12123 ts_tablet_manager.cc:1397] T 702190ed4e0242719e0fa48ebca06892 P c1817f7b66724d389d2ab38ab62a9be2: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.000s
I20260927 10:45:23.596777 12123 raft_consensus.cc:348] T 702190ed4e0242719e0fa48ebca06892 P c1817f7b66724d389d2ab38ab62a9be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c3c1aefa494b6cb07d9cff54ea0239" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40449 } } peers { permanent_uuid: "a8ca3a6958f44730b3951170de3fbbef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44079 } } peers { permanent_uuid: "c1817f7b66724d389d2ab38ab62a9be2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45875 } }
I20260927 10:45:23.597185 12123 raft_consensus.cc:374] T 702190ed4e0242719e0fa48ebca06892 P c1817f7b66724d389d2ab38ab62a9be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:23.597272 12123 raft_consensus.cc:729] T 702190ed4e0242719e0fa48ebca06892 P c1817f7b66724d389d2ab38ab62a9be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1817f7b66724d389d2ab38ab62a9be2, State: Initialized, Role: FOLLOWER
I20260927 10:45:23.597766 12123 consensus_queue.cc:260] T 702190ed4e0242719e0fa48ebca06892 P c1817f7b66724d389d2ab38ab62a9be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c3c1aefa494b6cb07d9cff54ea0239" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40449 } } peers { permanent_uuid: "a8ca3a6958f44730b3951170de3fbbef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44079 } } peers { permanent_uuid: "c1817f7b66724d389d2ab38ab62a9be2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45875 } }
I20260927 10:45:23.598496 12103 heartbeater.cc:503] Master 127.0.98.62:33567 was elected leader, sending a full tablet report...
I20260927 10:45:23.598985 12123 ts_tablet_manager.cc:1428] T 702190ed4e0242719e0fa48ebca06892 P c1817f7b66724d389d2ab38ab62a9be2: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260927 10:45:23.618337 12129 raft_consensus.cc:482] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:45:23.618602 12129 raft_consensus.cc:504] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c3c1aefa494b6cb07d9cff54ea0239" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40449 } } peers { permanent_uuid: "a8ca3a6958f44730b3951170de3fbbef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44079 } } peers { permanent_uuid: "c1817f7b66724d389d2ab38ab62a9be2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45875 } }
I20260927 10:45:23.619783 12129 leader_election.cc:288] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8ca3a6958f44730b3951170de3fbbef (127.0.98.1:44079), c1817f7b66724d389d2ab38ab62a9be2 (127.0.98.3:45875)
I20260927 10:45:23.625181 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.620142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37681 (local address 127.0.98.1:44079)
0927 10:45:23.620282 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:23.620290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:23.620535 (+ 245us) server_negotiation.cc:407] Connection header received
0927 10:45:23.620742 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:23.620745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:23.620801 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:23.620849 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:23.621291 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.622072 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.623706 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.624325 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.624402 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.624477 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:23.624969 (+ 492us) server_negotiation.cc:299] Negotiation successful
0927 10:45:23.625046 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:45:23.625181 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.620738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33249 (local address 127.0.98.3:45875)
0927 10:45:23.620859 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:23.620863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:23.621110 (+ 247us) server_negotiation.cc:407] Connection header received
0927 10:45:23.621283 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:23.621286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:23.621339 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:23.621385 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:23.621736 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.622452 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.623817 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.624395 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.624455 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.624509 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:23.624969 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 10:45:23.625046 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:45:23.625618 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.620400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45875 (local address 127.0.98.2:33249)
0927 10:45:23.621034 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:23.621049 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:23.621163 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:23.621507 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:23.621510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:23.621526 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:23.621628 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.621633 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.622596 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.622599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.623706 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.623713 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.624588 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.624591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.624786 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.624847 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 10:45:23.625388 (+ 541us) client_negotiation.cc:240] Negotiation successful
0927 10:45:23.625475 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":71,"thread_start_us":84,"threads_started":1}
I20260927 10:45:23.625890 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.619996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44079 (local address 127.0.98.2:37681)
0927 10:45:23.620432 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:23.620447 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:23.620617 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:23.621003 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:23.621005 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:23.621021 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:23.621171 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.621178 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.622199 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.622202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.623554 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.623564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.624588 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.624591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.624807 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.624850 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 10:45:23.625649 (+ 799us) client_negotiation.cc:240] Negotiation successful
0927 10:45:23.625731 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":69,"thread_start_us":137,"threads_started":1}
I20260927 10:45:23.626394 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702190ed4e0242719e0fa48ebca06892" candidate_uuid: "01c3c1aefa494b6cb07d9cff54ea0239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1817f7b66724d389d2ab38ab62a9be2" is_pre_election: true
I20260927 10:45:23.626638 11787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702190ed4e0242719e0fa48ebca06892" candidate_uuid: "01c3c1aefa494b6cb07d9cff54ea0239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8ca3a6958f44730b3951170de3fbbef" is_pre_election: true
I20260927 10:45:23.626781 12041 raft_consensus.cc:2454] T 702190ed4e0242719e0fa48ebca06892 P c1817f7b66724d389d2ab38ab62a9be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01c3c1aefa494b6cb07d9cff54ea0239 in term 0.
I20260927 10:45:23.626912 11787 raft_consensus.cc:2454] T 702190ed4e0242719e0fa48ebca06892 P a8ca3a6958f44730b3951170de3fbbef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01c3c1aefa494b6cb07d9cff54ea0239 in term 0.
I20260927 10:45:23.627287 11859 leader_election.cc:302] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c3c1aefa494b6cb07d9cff54ea0239, c1817f7b66724d389d2ab38ab62a9be2; no voters:
I20260927 10:45:23.627609 12129 raft_consensus.cc:2781] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:45:23.627713 12129 raft_consensus.cc:482] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:45:23.627756 12129 raft_consensus.cc:3037] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:23.629148 12129 raft_consensus.cc:504] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c3c1aefa494b6cb07d9cff54ea0239" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40449 } } peers { permanent_uuid: "a8ca3a6958f44730b3951170de3fbbef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44079 } } peers { permanent_uuid: "c1817f7b66724d389d2ab38ab62a9be2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45875 } }
I20260927 10:45:23.629520 12129 leader_election.cc:288] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239 [CANDIDATE]: Term 1 election: Requested vote from peers a8ca3a6958f44730b3951170de3fbbef (127.0.98.1:44079), c1817f7b66724d389d2ab38ab62a9be2 (127.0.98.3:45875)
I20260927 10:45:23.629981 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702190ed4e0242719e0fa48ebca06892" candidate_uuid: "01c3c1aefa494b6cb07d9cff54ea0239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1817f7b66724d389d2ab38ab62a9be2"
I20260927 10:45:23.630105 12041 raft_consensus.cc:3037] T 702190ed4e0242719e0fa48ebca06892 P c1817f7b66724d389d2ab38ab62a9be2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:23.630617 11787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702190ed4e0242719e0fa48ebca06892" candidate_uuid: "01c3c1aefa494b6cb07d9cff54ea0239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8ca3a6958f44730b3951170de3fbbef"
I20260927 10:45:23.630740 11787 raft_consensus.cc:3037] T 702190ed4e0242719e0fa48ebca06892 P a8ca3a6958f44730b3951170de3fbbef [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:23.632009 11787 raft_consensus.cc:2454] T 702190ed4e0242719e0fa48ebca06892 P a8ca3a6958f44730b3951170de3fbbef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01c3c1aefa494b6cb07d9cff54ea0239 in term 1.
I20260927 10:45:23.632411 12041 raft_consensus.cc:2454] T 702190ed4e0242719e0fa48ebca06892 P c1817f7b66724d389d2ab38ab62a9be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01c3c1aefa494b6cb07d9cff54ea0239 in term 1.
I20260927 10:45:23.632421 11859 leader_election.cc:302] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c3c1aefa494b6cb07d9cff54ea0239, a8ca3a6958f44730b3951170de3fbbef; no voters:
I20260927 10:45:23.632597 12129 raft_consensus.cc:2781] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:23.632784 12129 raft_consensus.cc:686] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239 [term 1 LEADER]: Becoming Leader. State: Replica: 01c3c1aefa494b6cb07d9cff54ea0239, State: Running, Role: LEADER
I20260927 10:45:23.633137 12129 consensus_queue.cc:237] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c3c1aefa494b6cb07d9cff54ea0239" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40449 } } peers { permanent_uuid: "a8ca3a6958f44730b3951170de3fbbef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44079 } } peers { permanent_uuid: "c1817f7b66724d389d2ab38ab62a9be2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45875 } }
I20260927 10:45:23.637001 11619 catalog_manager.cc:5495] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01c3c1aefa494b6cb07d9cff54ea0239 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "01c3c1aefa494b6cb07d9cff54ea0239" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01c3c1aefa494b6cb07d9cff54ea0239" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8ca3a6958f44730b3951170de3fbbef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1817f7b66724d389d2ab38ab62a9be2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45875 } health_report { overall_health: UNKNOWN } } }
I20260927 10:45:23.657670 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.652773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34020 (local address 127.0.98.2:40449)
0927 10:45:23.653132 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:23.653136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:23.653151 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:45:23.653237 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:23.653240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:23.653311 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:23.653428 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:45:23.653734 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.654616 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.655910 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.656247 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.656358 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.656693 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:45:23.657260 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:45:23.657389 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:23.657469 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 10:45:23.657528 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260927 10:45:23.665050 11787 raft_consensus.cc:1261] T 702190ed4e0242719e0fa48ebca06892 P a8ca3a6958f44730b3951170de3fbbef [term 1 FOLLOWER]: Refusing update from remote peer 01c3c1aefa494b6cb07d9cff54ea0239: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:45:23.665060 12041 raft_consensus.cc:1261] T 702190ed4e0242719e0fa48ebca06892 P c1817f7b66724d389d2ab38ab62a9be2 [term 1 FOLLOWER]: Refusing update from remote peer 01c3c1aefa494b6cb07d9cff54ea0239: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:45:23.665824 12133 consensus_queue.cc:1035] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1817f7b66724d389d2ab38ab62a9be2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:45:23.666373 12129 consensus_queue.cc:1035] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8ca3a6958f44730b3951170de3fbbef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 10:45:23.673781 12104 tablet.cc:2369] T 702190ed4e0242719e0fa48ebca06892 P c1817f7b66724d389d2ab38ab62a9be2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:45:23.676373 12136 mvcc.cc:205] Tried to move back new op lower bound from 7333912263322800128 to 7333912263205974016. Current Snapshot: MvccSnapshot[applied={T|T < 7333912263322800128}]
I20260927 10:45:23.681039 12137 mvcc.cc:205] Tried to move back new op lower bound from 7333912263322800128 to 7333912263205974016. Current Snapshot: MvccSnapshot[applied={T|T < 7333912263322800128}]
I20260927 10:45:23.689309 11619 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:45:23.689494 11619 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:45:23.691900 12138 mvcc.cc:205] Tried to move back new op lower bound from 7333912263322800128 to 7333912263205974016. Current Snapshot: MvccSnapshot[applied={T|T < 7333912263322800128}]
I20260927 10:45:23.693022 11619 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5200fb6a5f124d09aaba410f254001cb: Sending DeleteTablet for 3 replicas of tablet 702190ed4e0242719e0fa48ebca06892
I20260927 10:45:23.694175 11767 tablet_service.cc:1514] Processing DeleteTablet for tablet 702190ed4e0242719e0fa48ebca06892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58504
I20260927 10:45:23.694291 11902 tablet_service.cc:1514] Processing DeleteTablet for tablet 702190ed4e0242719e0fa48ebca06892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34006
I20260927 10:45:23.694547 12036 tablet_service.cc:1514] Processing DeleteTablet for tablet 702190ed4e0242719e0fa48ebca06892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60910
I20260927 10:45:23.694998 12146 tablet_replica.cc:330] T 702190ed4e0242719e0fa48ebca06892 P c1817f7b66724d389d2ab38ab62a9be2: stopping tablet replica
I20260927 10:45:23.695515 12146 raft_consensus.cc:2229] T 702190ed4e0242719e0fa48ebca06892 P c1817f7b66724d389d2ab38ab62a9be2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:45:23.695783 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 11700
I20260927 10:45:23.695832 12146 raft_consensus.cc:2258] T 702190ed4e0242719e0fa48ebca06892 P c1817f7b66724d389d2ab38ab62a9be2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:23.704548 12147 tablet_replica.cc:330] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239: stopping tablet replica
I20260927 10:45:23.705067 12147 raft_consensus.cc:2229] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:45:23.705638 12147 raft_consensus.cc:2258] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:23.708355 12147 ts_tablet_manager.cc:1905] T 702190ed4e0242719e0fa48ebca06892 P 01c3c1aefa494b6cb07d9cff54ea0239: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 10:45:23.710906 11606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44079 (error 108)
I20260927 10:45:23.711069 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 11839
I20260927 10:45:23.713629 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.713257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44079 (local address 127.0.0.1:58518)
0927 10:45:23.713427 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:23.713516 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 10:45:23.713946 11606 catalog_manager.cc:4584] TS a8ca3a6958f44730b3951170de3fbbef (127.0.98.1:44079): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 702190ed4e0242719e0fa48ebca06892: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44079: connect: Connection refused (error 111)
I20260927 10:45:23.714473 12146 ts_tablet_manager.cc:1905] T 702190ed4e0242719e0fa48ebca06892 P c1817f7b66724d389d2ab38ab62a9be2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:23.717135 12146 ts_tablet_manager.cc:1918] T 702190ed4e0242719e0fa48ebca06892 P c1817f7b66724d389d2ab38ab62a9be2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:45:23.717314 12146 log.cc:1196] T 702190ed4e0242719e0fa48ebca06892 P c1817f7b66724d389d2ab38ab62a9be2: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/702190ed4e0242719e0fa48ebca06892
I20260927 10:45:23.717739 12146 ts_tablet_manager.cc:1939] T 702190ed4e0242719e0fa48ebca06892 P c1817f7b66724d389d2ab38ab62a9be2: Deleting consensus metadata
I20260927 10:45:23.719300 11606 catalog_manager.cc:4841] TS c1817f7b66724d389d2ab38ab62a9be2 (127.0.98.3:45875): tablet 702190ed4e0242719e0fa48ebca06892 (table test-table [id=fe4abe98fc3a401c967ff7a0f2ea52f8]) successfully deleted
W20260927 10:45:23.719906 12120 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:23.719100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40449 (local address 127.0.0.1:34024)
0927 10:45:23.719281 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:23.719298 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:23.719387 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:23.719800 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40449: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":107}
I20260927 10:45:23.719993 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 11974
I20260927 10:45:23.727847 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 11587
I20260927 10:45:23.737601 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 10:45:24.082749 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 10:45:24.082909 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 10:45:24.428186 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 10:45:24.428 UTC [11173] LOG: received smart shutdown request
2026-09-27 10:45:24.432 UTC [11173] LOG: background worker "logical replication launcher" (PID 11182) exited with exit code 1
2026-09-27 10:45:24.432 UTC [11177] LOG: shutting down
2026-09-27 10:45:24.642 UTC [11173] LOG: database system is shut down
2026-09-27T10:45:24Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46542 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:45:24 dist-test-slave-dzhw krb5kdc[12158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:45:24 dist-test-slave-dzhw krb5kdc[12158](info): set up 2 sockets
Sep 27 10:45:24 dist-test-slave-dzhw krb5kdc[12158](info): commencing operation
krb5kdc: starting...
W20260927 10:45:25.816481 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:45:25 dist-test-slave-dzhw krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:45:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:45:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 10:45:25.994501 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskkqkM3n/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/postgres -l logfile start
2026-09-27 10:45:28.643 UTC [12188] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 10:45:28.643 UTC [12188] LOG: listening on IPv4 address "127.0.98.20", port 41455
2026-09-27 10:45:28.647 UTC [12188] LOG: listening on Unix socket "/tmp/.s.PGSQL.41455"
2026-09-27 10:45:28.681 UTC [12191] LOG: database system was shut down at 2026-09-27 10:45:27 UTC
2026-09-27 10:45:28.685 UTC [12188] LOG: database system is ready to accept connections
I20260927 10:45:29.638231 392 mini_postgres.cc:96] Postgres bound to 41455
2026-09-27 10:45:29.644 UTC [12199] FATAL: database "slave" does not exist
127.0.98.20:41455 - accepting connections
I20260927 10:45:29.645119 392 mini_ranger.cc:161] Starting Ranger...
I20260927 10:45:29.660036 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 10:45:30.174888 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 10:45:30.174981 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ranger-admin
2026-09-27 10:45:30,470 [I] DB FLAVOR :POSTGRES
2026-09-27 10:45:30,470 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 10:45:30,470 [I] Checking connection..
2026-09-27 10:45:30,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 10:45:30,737 [I] Checking connection passed.
2026-09-27 10:45:30,738 [I] --------- Verifying version history table ---------
2026-09-27 10:45:30,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:45:30,989 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 10:45:30,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:45:31,256 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 10:45:31,256 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 10:45:31,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 10:45:31.498 UTC [12263] WARNING: there is no transaction in progress
2026-09-27 10:45:31,507 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 10:45:31,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:45:31,788 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 10:45:31,789 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 10:45:31,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 10:45:32,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 10:45:32,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dzhw.c.gcp-upstream.internal','N') ;"
2026-09-27 10:45:32,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 10:45:32,782 [I] Table x_portal_user does not exist in database ranger
2026-09-27 10:45:32,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 10:45:33,041 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 10:45:33,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 10:45:33,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 10:45:33,510 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 10:45:33,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 10:45:34.457 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 10:45:34.474 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 10:45:34.490 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 10:45:34.506 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 10:45:34.522 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 10:45:34.539 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 10:45:34.594 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 10:45:34.601 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 10:45:34.616 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 10:45:34.627 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 10:45:34.637 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 10:45:34.651 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 10:45:34.661 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 10:45:34.667 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 10:45:34.675 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 10:45:34.683 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 10:45:35,349 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 10:45:35,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 10:45:35,608 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 10:45:35,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 10:45:35,848 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 10:45:35,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 10:45:36,099 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 10:45:36,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 10:45:36,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 10:45:36,591 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dzhw.c.gcp-upstream.internal';"
2026-09-27 10:45:36,843 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 10:45:36,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 10:45:37,073 [I] DB_PATCHES have already been applied
I20260927 10:45:37.079296 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskkqkM3n/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 10:45:37.079397 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 10:45:37.082784 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:53911
Sep 27, 2026 10:45:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:45:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:45:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 10:45:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 10:45:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 10:45:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 10:45:37 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 10:45:38.188861 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.544s user 0.000s sys 0.010s
Sep 27, 2026 10:45:58 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 10:45:58 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 10:45:58 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 10:45:58 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 10:45:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 10:45:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 10:45:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 10:45:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 10:45:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 10:45:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 10:45:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 10:45:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 10:45:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 10:45:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 10:45:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 10:45:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 10:45:58 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 10:45:59 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 10:46:00.145823 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 10:46:00 dist-test-slave-dzhw krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505960, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 10:46:00.235188 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 10:46:00.245326 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 10:46:01.192307 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 10:46:01.192394 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ranger-kms
I20260927 10:46:01.192411 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:41455
2026-09-27 10:46:01,383 [I] DB FLAVOR :POSTGRES
2026-09-27 10:46:01,384 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 10:46:01,384 [I] Checking connection
2026-09-27 10:46:01,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 10:46:01,641 [I] connection success
2026-09-27 10:46:01,641 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 10:46:01,641 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 10:46:01,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 10:46:01,957 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 10:46:01,957 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 10:46:01,957 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 10:46:01,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41455/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 10:46:02,252 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 10:46:02.696612 392 mini_ranger_kms.cc:310] Created kms service
I20260927 10:46:02.904155 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 10:46:02.951835 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 10:46:02 dist-test-slave-dzhw krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505960, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 10:46:03.356895 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 10:46:03.357048 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskkqkM3n/build/debug/bin/postgresql.jar:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskkqkM3n/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 10:46:03.357069 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 10:46:03.361476 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12551
I20260927 10:46:03.361588 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:60235
Sep 27, 2026 10:46:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:46:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:46:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 10:46:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 10:46:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 10:46:05 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 10:46:05.662671 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.427s user 0.000s sys 0.174s
I20260927 10:46:05.662802 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 10:46:05.662889 392 mini_ranger_kms.cc:401] 127.0.98.20:60235/kms/v1/keys
Sep 27 10:46:05 dist-test-slave-dzhw krb5kdc[12158](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790505965, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 10:46:06 dist-test-slave-dzhw krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505965, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 10:46:06 dist-test-slave-dzhw krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505965, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 10:46:08 dist-test-slave-dzhw krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:46:08.834291 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43293
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43293
--ranger_config_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60235/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:46:08.969939 12602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:08.970225 12602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:08.970265 12602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:08.974583 12602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:46:08.974651 12602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:08.974670 12602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:08.974686 12602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:46:08.974725 12602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:46:08.980402 12602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:60235/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43293
--ranger_config_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:08.981595 12602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:08.982708 12602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:08.993670 12607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:08.993523 12602 server_base.cc:1034] running on GCE node
W20260927 10:46:08.993832 12610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:08.993541 12608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:08.994699 12602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:08.995927 12602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:08.997159 12602 hybrid_clock.cc:648] HybridClock initialized: now 1790505968997131 us; error 52 us; skew 500 ppm
Sep 27 10:46:09 dist-test-slave-dzhw krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:09.007776 12602 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:46:09.008946 12602 webserver.cc:460] Webserver started at http://127.0.98.62:39509/ using document root <none> and password file <none>
I20260927 10:46:09.009549 12602 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:09.009606 12602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:09.009838 12602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:46:09 dist-test-slave-dzhw krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 10:46:09.059212 12602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "ba620296e76d440ab0f90e22bc347339"
format_stamp: "Formatted at 2026-09-27 10:46:09 on dist-test-slave-dzhw"
server_key: "79337b2b9857a4c8b245304730182180"
server_key_iv: "925f5741a1c1e724c0847defbba3961f"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:09.059829 12602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "ba620296e76d440ab0f90e22bc347339"
format_stamp: "Formatted at 2026-09-27 10:46:09 on dist-test-slave-dzhw"
server_key: "79337b2b9857a4c8b245304730182180"
server_key_iv: "925f5741a1c1e724c0847defbba3961f"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:09.063968 12602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:46:09.078145 12617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:09.079648 12602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:46:09.079816 12602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "ba620296e76d440ab0f90e22bc347339"
format_stamp: "Formatted at 2026-09-27 10:46:09 on dist-test-slave-dzhw"
server_key: "79337b2b9857a4c8b245304730182180"
server_key_iv: "925f5741a1c1e724c0847defbba3961f"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:09.079964 12602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:09.103670 12602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:09.106679 12602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:09.106892 12602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:09.115070 12602 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43293
I20260927 10:46:09.115077 12669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43293 every 8 connection(s)
I20260927 10:46:09.116171 12602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:46:09.119058 12670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:46:09.123569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 12602
I20260927 10:46:09.123718 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:46:09.124092 392 external_mini_cluster.cc:1428] Setting key 53195101b27d8ee2986f1a6d1a320baa
I20260927 10:46:09.124888 12670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339: Bootstrap starting.
I20260927 10:46:09.127689 12670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339: Neither blocks nor log segments found. Creating new log.
I20260927 10:46:09.128647 12670 log.cc:824] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:09.130985 12670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339: No bootstrap required, opened a new log
Sep 27 10:46:09 dist-test-slave-dzhw krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:46:09.135255 12670 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba620296e76d440ab0f90e22bc347339" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43293 } }
I20260927 10:46:09.135507 12670 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:46:09.135578 12670 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba620296e76d440ab0f90e22bc347339, State: Initialized, Role: FOLLOWER
I20260927 10:46:09.136113 12670 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba620296e76d440ab0f90e22bc347339" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43293 } }
I20260927 10:46:09.136299 12670 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:46:09.136405 12670 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:46:09.136520 12670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:46:09.137806 12670 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba620296e76d440ab0f90e22bc347339" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43293 } }
I20260927 10:46:09.138195 12670 leader_election.cc:302] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba620296e76d440ab0f90e22bc347339; no voters:
I20260927 10:46:09.138680 12670 leader_election.cc:288] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:46:09.138837 12675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:46:09.139071 12675 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [term 1 LEADER]: Becoming Leader. State: Replica: ba620296e76d440ab0f90e22bc347339, State: Running, Role: LEADER
I20260927 10:46:09.139426 12675 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba620296e76d440ab0f90e22bc347339" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43293 } }
I20260927 10:46:09.140060 12670 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:46:09.140975 12675 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba620296e76d440ab0f90e22bc347339" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba620296e76d440ab0f90e22bc347339" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43293 } } }
I20260927 10:46:09.141198 12675 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [sys.catalog]: This master's current role is: LEADER
I20260927 10:46:09.141455 12675 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba620296e76d440ab0f90e22bc347339. Latest consensus state: current_term: 1 leader_uuid: "ba620296e76d440ab0f90e22bc347339" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba620296e76d440ab0f90e22bc347339" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43293 } } }
I20260927 10:46:09.141594 12675 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [sys.catalog]: This master's current role is: LEADER
I20260927 10:46:09.141877 12673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:09.125664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43566 (local address 127.0.98.62:43293)
0927 10:46:09.126200 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:09.126219 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:09.126264 (+ 45us) server_negotiation.cc:407] Connection header received
0927 10:46:09.127103 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:09.127126 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:09.127540 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:09.128885 (+ 1345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:09.129527 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:09.130950 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:09.132145 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:09.132931 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:09.133109 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:09.136050 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:09.136056 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:09.136073 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:09.136105 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:09.138182 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:09.138724 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:09.138730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:09.138737 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:09.138883 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:09.139212 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:09.139216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:09.139217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:09.139713 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:09.140460 (+ 747us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:09.140606 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 10:46:09.140888 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":142,"threads_started":1}
I20260927 10:46:09.141935 12684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:46:09.142334 12670 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 10:46:09.149291 12684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:46:09.156905 12684 catalog_manager.cc:1347] Generated new cluster ID: f97498f7621c49eea16a3aa152ee470a
I20260927 10:46:09.156980 12684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:46:09.172317 12684 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:46:09.173733 12684 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:46:09.179965 12684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339: Generated new TSK 0
I20260927 10:46:09.180923 12684 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 10:46:09.981513 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60235/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43293
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:46:10.139222 12715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:10.139660 12715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:10.139768 12715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:10.145910 12715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:10.146099 12715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:10.146229 12715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:46:10.153981 12715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:60235/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:10.155633 12715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:10.157004 12715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:10.171916 12725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:10.172456 12723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:10.173969 12715 server_base.cc:1034] running on GCE node
W20260927 10:46:10.174144 12722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:10.174687 12715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:10.175479 12715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:10.176654 12715 hybrid_clock.cc:648] HybridClock initialized: now 1790505970176626 us; error 46 us; skew 500 ppm
Sep 27 10:46:10 dist-test-slave-dzhw krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:10.180491 12715 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 10:46:10.181584 12715 webserver.cc:460] Webserver started at http://127.0.98.1:43799/ using document root <none> and password file <none>
I20260927 10:46:10.182327 12715 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:10.182479 12715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:10.182817 12715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:46:10 dist-test-slave-dzhw krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 10:46:10.201297 12715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "0a860b4df99b48afa8a3634737a191aa"
format_stamp: "Formatted at 2026-09-27 10:46:10 on dist-test-slave-dzhw"
server_key: "aea82fd637016259b3557b8a18b84a07"
server_key_iv: "aa093ba807a252ec72dfae52cee05592"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:10.202152 12715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0a860b4df99b48afa8a3634737a191aa"
format_stamp: "Formatted at 2026-09-27 10:46:10 on dist-test-slave-dzhw"
server_key: "aea82fd637016259b3557b8a18b84a07"
server_key_iv: "aa093ba807a252ec72dfae52cee05592"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:10.212185 12715 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.000s sys 0.003s
I20260927 10:46:10.238874 12732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:10.240746 12715 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.002s
I20260927 10:46:10.240892 12715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "0a860b4df99b48afa8a3634737a191aa"
format_stamp: "Formatted at 2026-09-27 10:46:10 on dist-test-slave-dzhw"
server_key: "aea82fd637016259b3557b8a18b84a07"
server_key_iv: "aa093ba807a252ec72dfae52cee05592"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:10.241036 12715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:10.281036 12715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:10.284411 12715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:10.284608 12715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:10.285236 12715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:46:10.286125 12715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:46:10.286193 12715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:10.286314 12715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:46:10.286367 12715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:10.297294 12715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34397
I20260927 10:46:10.297313 12847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34397 every 8 connection(s)
I20260927 10:46:10.298326 12715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:46:10.302229 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 12715
I20260927 10:46:10.302371 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:46:10.302709 392 external_mini_cluster.cc:1428] Setting key 848205fc1d2b4873997f51a03292602d
Sep 27 10:46:10 dist-test-slave-dzhw krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 10:46:10.315506 12850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:10.300279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33371 (local address 127.0.98.62:43293)
0927 10:46:10.301659 (+ 1380us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:10.301664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:10.301681 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:46:10.302522 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:10.302525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:10.302590 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:10.302675 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:10.304375 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:10.305207 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:10.307084 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:10.307493 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:10.307608 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:10.310898 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:10.310905 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:10.310908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:10.310939 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:10.312910 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:10.313590 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:10.313596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:10.313598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:10.313691 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:10.314134 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:10.314137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:10.314139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:10.314331 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:10.314473 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:10.315174 (+ 701us) server_negotiation.cc:299] Negotiation successful
0927 10:46:10.315316 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1302,"thread_start_us":79,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 10:46:10.316612 12851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:10.300690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43293 (local address 127.0.98.1:33371)
0927 10:46:10.301147 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:10.301199 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:10.302149 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:10.302991 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:10.303000 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:10.303542 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:10.304199 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:10.304232 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:10.305354 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:10.305360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:10.306926 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:10.306940 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:10.307806 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:10.307812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:10.308050 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:10.308773 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:10.308810 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:10.310658 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:10.313075 (+ 2417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:10.313083 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:10.313106 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:10.313463 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:10.313820 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:10.313823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:10.313826 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:10.314018 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:10.314480 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:10.314489 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:10.314932 (+ 443us) client_negotiation.cc:769] Sending connection context
0927 10:46:10.315137 (+ 205us) client_negotiation.cc:240] Negotiation successful
0927 10:46:10.315463 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":139,"threads_started":1}
I20260927 10:46:10.318316 12848 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43293
I20260927 10:46:10.318676 12848 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:10.319298 12848 heartbeater.cc:511] Master 127.0.98.62:43293 requested a full tablet report, sending...
I20260927 10:46:10.321035 12634 ts_manager.cc:194] Registered new tserver with Master: 0a860b4df99b48afa8a3634737a191aa (127.0.98.1:34397)
I20260927 10:46:10.322886 12634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33371
Sep 27 10:46:10 dist-test-slave-dzhw krb5kdc[12158](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790505970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 10:46:10.334730 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60235/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43293
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:46:10.464977 12854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:10.465417 12854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:10.465530 12854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:10.470103 12854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:10.470211 12854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:10.470341 12854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:46:10.476362 12854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:60235/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:10.477748 12854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:10.479048 12854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:10.489995 12861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:10.489979 12860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:10.490394 12863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:10.490988 12854 server_base.cc:1034] running on GCE node
I20260927 10:46:10.491966 12854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:10.492797 12854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:10.493993 12854 hybrid_clock.cc:648] HybridClock initialized: now 1790505970493948 us; error 69 us; skew 500 ppm
Sep 27 10:46:10 dist-test-slave-dzhw krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:10.497591 12854 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 10:46:10.498695 12854 webserver.cc:460] Webserver started at http://127.0.98.2:37269/ using document root <none> and password file <none>
I20260927 10:46:10.499285 12854 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:10.499384 12854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:10.499625 12854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:46:10 dist-test-slave-dzhw krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 10:46:10.520546 12854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "fcc6bbd127b24986958f2053b6fc8cb7"
format_stamp: "Formatted at 2026-09-27 10:46:10 on dist-test-slave-dzhw"
server_key: "264168772472d69674b1c79ab1e99356"
server_key_iv: "d2fae6f0a3cdffcfb8c4a6e4017a0ae8"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:10.521538 12854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fcc6bbd127b24986958f2053b6fc8cb7"
format_stamp: "Formatted at 2026-09-27 10:46:10 on dist-test-slave-dzhw"
server_key: "264168772472d69674b1c79ab1e99356"
server_key_iv: "d2fae6f0a3cdffcfb8c4a6e4017a0ae8"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:10.531306 12854 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260927 10:46:10.552299 12870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:10.554096 12854 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260927 10:46:10.554303 12854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "fcc6bbd127b24986958f2053b6fc8cb7"
format_stamp: "Formatted at 2026-09-27 10:46:10 on dist-test-slave-dzhw"
server_key: "264168772472d69674b1c79ab1e99356"
server_key_iv: "d2fae6f0a3cdffcfb8c4a6e4017a0ae8"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:10.554551 12854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:10.576423 12854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:10.582346 12854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:10.582599 12854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:10.583400 12854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:46:10.584618 12854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:46:10.584684 12854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:10.584789 12854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:46:10.584826 12854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:10.596767 12854 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44915
I20260927 10:46:10.596884 12983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44915 every 8 connection(s)
I20260927 10:46:10.598178 12854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:46:10.602873 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 12854
I20260927 10:46:10.603001 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:46:10.603317 392 external_mini_cluster.cc:1428] Setting key 0c6b425d0e58fcbc5e9bedb09bc3b97c
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 27 10:46:10 dist-test-slave-dzhw krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:46:10.623384 12850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:10.600410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51425 (local address 127.0.98.62:43293)
0927 10:46:10.600555 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:10.600560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:10.601673 (+ 1113us) server_negotiation.cc:407] Connection header received
0927 10:46:10.604676 (+ 3003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:10.604680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:10.604733 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:10.604783 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:10.607337 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:10.608161 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:10.614664 (+ 6503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:10.615048 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:10.615180 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:10.618756 (+ 3576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:10.618764 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:10.618770 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:10.618812 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:10.620490 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:10.621213 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:10.621219 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:10.621221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:10.621320 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:10.621783 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:10.621790 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:10.621795 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:10.622079 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:10.622195 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:10.622945 (+ 750us) server_negotiation.cc:299] Negotiation successful
0927 10:46:10.623129 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:46:10.624600 12986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:10.600936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43293 (local address 127.0.98.2:51425)
0927 10:46:10.601505 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:10.601561 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:10.604424 (+ 2863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:10.605953 (+ 1529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:10.605963 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:10.606375 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:10.607136 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:10.607158 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:10.608337 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:10.608340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:10.609784 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:10.609795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:10.615283 (+ 5488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:10.615288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:10.615496 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:10.616142 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:10.616160 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:10.618501 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:10.620626 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:10.620633 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:10.620655 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:10.621048 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:10.621431 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:10.621434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:10.621436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:10.621644 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:10.622202 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:10.622209 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:10.622620 (+ 411us) client_negotiation.cc:769] Sending connection context
0927 10:46:10.622844 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 10:46:10.623444 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":152,"threads_started":1}
I20260927 10:46:10.626344 12984 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43293
I20260927 10:46:10.626650 12984 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:10.627285 12984 heartbeater.cc:511] Master 127.0.98.62:43293 requested a full tablet report, sending...
I20260927 10:46:10.628535 12634 ts_manager.cc:194] Registered new tserver with Master: fcc6bbd127b24986958f2053b6fc8cb7 (127.0.98.2:44915)
I20260927 10:46:10.629356 12634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 10:46:10.637562 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60235/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43293
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:46:10.803936 12989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:10.804391 12989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:10.804491 12989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:10.810949 12989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:10.811091 12989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:10.811228 12989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
Sep 27 10:46:10 dist-test-slave-dzhw krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 10:46:10.819669 12989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:60235/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:10.821310 12989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:10.822688 12989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:10.836386 12995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:10.837630 12989 server_base.cc:1034] running on GCE node
W20260927 10:46:10.841188 12997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:10.843104 12994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:10.843830 12989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:10.844696 12989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:10.845996 12989 hybrid_clock.cc:648] HybridClock initialized: now 1790505970845907 us; error 101 us; skew 500 ppm
Sep 27 10:46:10 dist-test-slave-dzhw krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:10.849571 12989 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 10:46:10.850658 12989 webserver.cc:460] Webserver started at http://127.0.98.3:43117/ using document root <none> and password file <none>
I20260927 10:46:10.851377 12989 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:10.851490 12989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:10.851758 12989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:46:10 dist-test-slave-dzhw krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 10:46:10.893006 12989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "491994d1f4d84b31b1515a52c3fd084d"
format_stamp: "Formatted at 2026-09-27 10:46:10 on dist-test-slave-dzhw"
server_key: "4f0144f252b98380103b866f079d75ed"
server_key_iv: "72cd759dbf66e64c96ebd8da65e89715"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:10.893960 12989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "491994d1f4d84b31b1515a52c3fd084d"
format_stamp: "Formatted at 2026-09-27 10:46:10 on dist-test-slave-dzhw"
server_key: "4f0144f252b98380103b866f079d75ed"
server_key_iv: "72cd759dbf66e64c96ebd8da65e89715"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:10.899480 12989 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260927 10:46:10.928923 13004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:10.930685 12989 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.001s
I20260927 10:46:10.930845 12989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "491994d1f4d84b31b1515a52c3fd084d"
format_stamp: "Formatted at 2026-09-27 10:46:10 on dist-test-slave-dzhw"
server_key: "4f0144f252b98380103b866f079d75ed"
server_key_iv: "72cd759dbf66e64c96ebd8da65e89715"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:10.931000 12989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:10.949294 12989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:10.952579 12989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:10.952795 12989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:10.953403 12989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:46:10.954232 12989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:46:10.954303 12989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:10.954377 12989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:46:10.954421 12989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:10.964737 12989 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36081
I20260927 10:46:10.964824 13117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36081 every 8 connection(s)
I20260927 10:46:10.965744 12989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:46:10.970148 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 12989
I20260927 10:46:10.970271 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:46:10.970597 392 external_mini_cluster.cc:1428] Setting key 652b6ed87893a9aa3a11ac452db75fc7
Sep 27 10:46:10 dist-test-slave-dzhw krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:46:10.983194 12850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:10.969529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41291 (local address 127.0.98.62:43293)
0927 10:46:10.969690 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:10.969695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:10.969713 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:46:10.970865 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:10.970869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:10.970922 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:10.970975 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:10.972733 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:10.973557 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:10.975325 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:10.975662 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:10.975754 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:10.978879 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:10.978906 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:10.978909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:10.978935 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:10.980689 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:10.981311 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:10.981315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:10.981317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:10.981404 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:10.981810 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:10.981814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:10.981815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:10.982048 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:10.982156 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:10.982886 (+ 730us) server_negotiation.cc:299] Negotiation successful
0927 10:46:10.983018 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 10:46:10.984553 13120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:10.968439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43293 (local address 127.0.98.3:41291)
0927 10:46:10.969257 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:10.969320 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:10.970595 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:10.971566 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:10.971575 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:10.971933 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:10.972571 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:10.972588 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:10.973723 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:10.973726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:10.975176 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:10.975188 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:10.975956 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:10.975960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:10.976155 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:10.976879 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:10.976901 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:10.978671 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:10.980822 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:10.980828 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:10.980845 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:10.981181 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:10.981501 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:10.981503 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:10.981505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:10.981684 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:10.982169 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:10.982176 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:10.982551 (+ 375us) client_negotiation.cc:769] Sending connection context
0927 10:46:10.982783 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 10:46:10.983108 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":155,"threads_started":1}
I20260927 10:46:10.986148 13118 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43293
I20260927 10:46:10.986406 13118 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:10.987025 13118 heartbeater.cc:511] Master 127.0.98.62:43293 requested a full tablet report, sending...
I20260927 10:46:10.988250 12634 ts_manager.cc:194] Registered new tserver with Master: 491994d1f4d84b31b1515a52c3fd084d (127.0.98.3:36081)
I20260927 10:46:10.989018 12634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41291
I20260927 10:46:10.997118 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:46:10.997902 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 12715
I20260927 10:46:11.006441 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 12854
I20260927 10:46:11.013157 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 12989
I20260927 10:46:11.021765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 12602
I20260927 10:46:11.029913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43293
--webserver_interface=127.0.98.62
--webserver_port=39509
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43293
--ranger_config_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60235/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60235/kms with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:46:11.139520 13121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:11.139827 13121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:11.139900 13121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:11.143862 13121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:46:11.143960 13121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:11.143998 13121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:11.144035 13121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:46:11.144070 13121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:46:11.149634 13121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60235/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43293
--ranger_config_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:11.150715 13121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:11.151708 13121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:11.160542 13127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:11.160542 13129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:11.160533 13126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:11.160876 13121 server_base.cc:1034] running on GCE node
I20260927 10:46:11.161419 13121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:11.162564 13121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:11.163723 13121 hybrid_clock.cc:648] HybridClock initialized: now 1790505971163712 us; error 24 us; skew 500 ppm
Sep 27 10:46:11 dist-test-slave-dzhw krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:11.166703 13121 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:46:11.167574 13121 webserver.cc:460] Webserver started at http://127.0.98.62:39509/ using document root <none> and password file <none>
I20260927 10:46:11.168112 13121 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:11.168177 13121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 10:46:11 dist-test-slave-dzhw krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 10:46:11.232378 13121 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 10:46:11.234838 13137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:11.236214 13121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:46:11.236342 13121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "ba620296e76d440ab0f90e22bc347339"
format_stamp: "Formatted at 2026-09-27 10:46:09 on dist-test-slave-dzhw"
server_key: "79337b2b9857a4c8b245304730182180"
server_key_iv: "925f5741a1c1e724c0847defbba3961f"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:11.236914 13121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:11.285676 13121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:11.288897 13121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:11.289106 13121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:11.296489 13121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43293
I20260927 10:46:11.296510 13189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43293 every 8 connection(s)
I20260927 10:46:11.297552 13121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:46:11.298352 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 13121
I20260927 10:46:11.298851 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60235/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34397
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43293
--webserver_port=43799
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60235/kms with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:46:11.302721 13190 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:46:11.304324 13190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339: Bootstrap starting.
I20260927 10:46:11.320495 13190 log.cc:824] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:11.327019 13190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:46:11.327442 13190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339: Bootstrap complete.
I20260927 10:46:11.331621 13190 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba620296e76d440ab0f90e22bc347339" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43293 } }
I20260927 10:46:11.332006 13190 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba620296e76d440ab0f90e22bc347339, State: Initialized, Role: FOLLOWER
I20260927 10:46:11.332486 13190 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba620296e76d440ab0f90e22bc347339" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43293 } }
I20260927 10:46:11.332618 13190 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:46:11.332695 13190 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:46:11.332798 13190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:46:11.334805 13190 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba620296e76d440ab0f90e22bc347339" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43293 } }
I20260927 10:46:11.335146 13190 leader_election.cc:302] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba620296e76d440ab0f90e22bc347339; no voters:
I20260927 10:46:11.335494 13190 leader_election.cc:288] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 10:46:11.335638 13194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 10:46:11.335881 13194 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [term 2 LEADER]: Becoming Leader. State: Replica: ba620296e76d440ab0f90e22bc347339, State: Running, Role: LEADER
I20260927 10:46:11.336187 13194 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba620296e76d440ab0f90e22bc347339" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43293 } }
I20260927 10:46:11.336741 13190 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:46:11.337175 13195 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba620296e76d440ab0f90e22bc347339. Latest consensus state: current_term: 2 leader_uuid: "ba620296e76d440ab0f90e22bc347339" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba620296e76d440ab0f90e22bc347339" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43293 } } }
I20260927 10:46:11.337297 13195 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [sys.catalog]: This master's current role is: LEADER
I20260927 10:46:11.337648 13196 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ba620296e76d440ab0f90e22bc347339" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba620296e76d440ab0f90e22bc347339" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43293 } } }
I20260927 10:46:11.337745 13196 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339 [sys.catalog]: This master's current role is: LEADER
I20260927 10:46:11.338028 13198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:46:11.341359 13198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:46:11.341907 13190 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 10:46:11.342013 13198 catalog_manager.cc:1259] Loaded cluster ID: f97498f7621c49eea16a3aa152ee470a
I20260927 10:46:11.342059 13198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:46:11.344812 13198 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:46:11.345481 13198 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339: Loaded TSK: 0
I20260927 10:46:11.346745 13198 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:46:11.438486 13192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:11.438795 13192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:11.438871 13192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:11.443679 13192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:11.443742 13192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:11.443853 13192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:46:11.449308 13192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60235/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43799
--tserver_master_addrs=127.0.98.62:43293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:11.450371 13192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:11.451416 13192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:11.463325 13230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:11.463421 13227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:11.463268 13228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:11.464715 13192 server_base.cc:1034] running on GCE node
I20260927 10:46:11.465152 13192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:11.465878 13192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:11.467067 13192 hybrid_clock.cc:648] HybridClock initialized: now 1790505971467034 us; error 44 us; skew 500 ppm
Sep 27 10:46:11 dist-test-slave-dzhw krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:11.470148 13192 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 10:46:11.471264 13192 webserver.cc:460] Webserver started at http://127.0.98.1:43799/ using document root <none> and password file <none>
I20260927 10:46:11.471866 13192 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:11.471977 13192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 10:46:11 dist-test-slave-dzhw krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 10:46:11.494690 13192 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:46:11.497856 13238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:11.499603 13192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:46:11.499760 13192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "0a860b4df99b48afa8a3634737a191aa"
format_stamp: "Formatted at 2026-09-27 10:46:10 on dist-test-slave-dzhw"
server_key: "aea82fd637016259b3557b8a18b84a07"
server_key_iv: "aa093ba807a252ec72dfae52cee05592"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:11.500602 13192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:11.533901 13192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:11.537521 13192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:11.537825 13192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:11.538651 13192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:46:11.539681 13192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:46:11.539747 13192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:11.539813 13192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:46:11.539840 13192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:11.550902 13192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34397
I20260927 10:46:11.551095 13351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34397 every 8 connection(s)
I20260927 10:46:11.552235 13192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:46:11.558854 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 13192
I20260927 10:46:11.559355 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60235/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44915
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43293
--webserver_port=37269
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60235/kms with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:46:11 dist-test-slave-dzhw krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:46:11.584580 13354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:11.554990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50251 (local address 127.0.98.62:43293)
0927 10:46:11.555550 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:11.555573 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:11.556760 (+ 1187us) server_negotiation.cc:407] Connection header received
0927 10:46:11.558455 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:11.558471 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:11.558891 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:11.559272 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:11.560775 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:11.562330 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:11.564216 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:11.564989 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:11.565137 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:11.578085 (+ 12948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:11.578095 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:11.578117 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:11.578161 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:11.580432 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:11.581198 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:11.581206 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:11.581211 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:11.581366 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:11.581813 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:11.581817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:11.581818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:11.582235 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:11.582382 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:11.583128 (+ 746us) server_negotiation.cc:299] Negotiation successful
0927 10:46:11.583472 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":115,"threads_started":1}
I20260927 10:46:11.586192 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:11.556002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43293 (local address 127.0.98.1:50251)
0927 10:46:11.556549 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:11.556608 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:11.557913 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:11.559848 (+ 1935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:11.559855 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:11.560122 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:11.560588 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:11.560601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:11.562542 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:11.562548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:11.564049 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:11.564064 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:11.565426 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:11.565431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:11.565621 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:11.574759 (+ 9138us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:11.574787 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:11.577740 (+ 2953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:11.580624 (+ 2884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:11.580631 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:11.580648 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:11.581040 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:11.581494 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:11.581497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:11.581499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:11.581696 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:11.582367 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:11.582373 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:11.582744 (+ 371us) client_negotiation.cc:769] Sending connection context
0927 10:46:11.584960 (+ 2216us) client_negotiation.cc:240] Negotiation successful
0927 10:46:11.585298 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":144,"threads_started":1}
I20260927 10:46:11.590111 13352 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43293
I20260927 10:46:11.590385 13352 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:11.591020 13352 heartbeater.cc:511] Master 127.0.98.62:43293 requested a full tablet report, sending...
W20260927 10:46:11.593313 13352 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43293 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 10:46:11.593439 13352 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:11.593650 13352 heartbeater.cc:511] Master 127.0.98.62:43293 requested a full tablet report, sending...
I20260927 10:46:11.594517 13352 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:11.594697 13352 heartbeater.cc:511] Master 127.0.98.62:43293 requested a full tablet report, sending...
W20260927 10:46:11.595403 13352 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:46:11.709424 13356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:11.709784 13356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:11.709860 13356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:11.714727 13356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:11.714824 13356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:11.714936 13356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:46:11.720726 13356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60235/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44915
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37269
--tserver_master_addrs=127.0.98.62:43293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:11.721935 13356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:11.723114 13356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:11.737439 13362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:11.738173 13361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:11.738183 13364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:11.739714 13356 server_base.cc:1034] running on GCE node
I20260927 10:46:11.740303 13356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:11.741428 13356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:11.742627 13356 hybrid_clock.cc:648] HybridClock initialized: now 1790505971742586 us; error 48 us; skew 500 ppm
Sep 27 10:46:11 dist-test-slave-dzhw krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:11.747009 13356 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 10:46:11.748330 13356 webserver.cc:460] Webserver started at http://127.0.98.2:37269/ using document root <none> and password file <none>
I20260927 10:46:11.749109 13356 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:11.749223 13356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 10:46:11 dist-test-slave-dzhw krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 10:46:11.778118 13356 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 10:46:11.781291 13372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:11.782935 13356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:46:11.783070 13356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "fcc6bbd127b24986958f2053b6fc8cb7"
format_stamp: "Formatted at 2026-09-27 10:46:10 on dist-test-slave-dzhw"
server_key: "264168772472d69674b1c79ab1e99356"
server_key_iv: "d2fae6f0a3cdffcfb8c4a6e4017a0ae8"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:11.783810 13356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:11.825726 13356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:11.829665 13356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:11.829893 13356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:11.830703 13356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:46:11.832059 13356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:46:11.832139 13356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:11.832213 13356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:46:11.832247 13356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:11.843792 13356 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44915
I20260927 10:46:11.843845 13485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44915 every 8 connection(s)
I20260927 10:46:11.845134 13356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:46:11.851625 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 13356
I20260927 10:46:11.852104 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60235/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36081
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43293
--webserver_port=43117
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60235/kms with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:46:11 dist-test-slave-dzhw krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:46:11.862390 13354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:11.847297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43471 (local address 127.0.98.62:43293)
0927 10:46:11.847459 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:11.847464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:11.848626 (+ 1162us) server_negotiation.cc:407] Connection header received
0927 10:46:11.849801 (+ 1175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:11.849806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:11.849869 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:11.849963 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:11.851546 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:11.852366 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:11.854150 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:11.854527 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:11.854612 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:11.857770 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:11.857775 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:11.857778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:11.857813 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:11.859614 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:11.860329 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:11.860333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:11.860335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:11.860434 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:11.860893 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:11.860897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:11.860898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:11.861140 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:11.861223 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:11.862057 (+ 834us) server_negotiation.cc:299] Negotiation successful
0927 10:46:11.862211 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:46:11.864185 13488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:11.847856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43293 (local address 127.0.98.2:43471)
0927 10:46:11.848418 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:11.848474 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:11.849546 (+ 1072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:11.850292 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:11.850301 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:11.850686 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:11.851364 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:11.851387 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:11.852539 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:11.852542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:11.854005 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:11.854016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:11.854846 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:11.854850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:11.855056 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:11.855795 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:11.855816 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:11.857527 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:11.859776 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:11.859783 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:11.859804 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:11.860199 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:11.860558 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:11.860560 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:11.860562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:11.860771 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:11.861316 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:11.861323 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:11.861738 (+ 415us) client_negotiation.cc:769] Sending connection context
0927 10:46:11.862684 (+ 946us) client_negotiation.cc:240] Negotiation successful
0927 10:46:11.863071 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":157,"threads_started":1}
I20260927 10:46:11.865651 13486 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43293
I20260927 10:46:11.865931 13486 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:11.866616 13486 heartbeater.cc:511] Master 127.0.98.62:43293 requested a full tablet report, sending...
W20260927 10:46:11.868614 13486 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43293 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 10:46:11.868754 13486 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:11.868956 13486 heartbeater.cc:511] Master 127.0.98.62:43293 requested a full tablet report, sending...
I20260927 10:46:11.869952 13486 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:11.870143 13486 heartbeater.cc:511] Master 127.0.98.62:43293 requested a full tablet report, sending...
W20260927 10:46:11.871029 13486 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:46:11.993295 13489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:11.993608 13489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:11.993708 13489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:11.998210 13489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:11.998302 13489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:11.998436 13489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:46:12.005370 13489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60235/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43117
--tserver_master_addrs=127.0.98.62:43293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:12.006817 13489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:12.008178 13489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:12.017875 13495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:12.017783 13496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:12.017795 13498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:12.018396 13489 server_base.cc:1034] running on GCE node
I20260927 10:46:12.018900 13489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:12.019645 13489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:12.027074 13489 hybrid_clock.cc:648] HybridClock initialized: now 1790505972027021 us; error 61 us; skew 500 ppm
Sep 27 10:46:12 dist-test-slave-dzhw krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:12.031172 13489 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 10:46:12.032372 13489 webserver.cc:460] Webserver started at http://127.0.98.3:43117/ using document root <none> and password file <none>
I20260927 10:46:12.033113 13489 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:12.033219 13489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 10:46:12 dist-test-slave-dzhw krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 10:46:12.052874 13489 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 10:46:12.056365 13506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:12.058068 13489 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 10:46:12.058280 13489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "491994d1f4d84b31b1515a52c3fd084d"
format_stamp: "Formatted at 2026-09-27 10:46:10 on dist-test-slave-dzhw"
server_key: "4f0144f252b98380103b866f079d75ed"
server_key_iv: "72cd759dbf66e64c96ebd8da65e89715"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:12.059082 13489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:12.092370 13489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:12.096170 13489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:12.096392 13489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:12.097147 13489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:46:12.098237 13489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:46:12.098299 13489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:12.098356 13489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:46:12.098397 13489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:12.111057 13489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36081
I20260927 10:46:12.112452 13489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:46:12.114639 13619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36081 every 8 connection(s)
I20260927 10:46:12.119602 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 13489
Sep 27 10:46:12 dist-test-slave-dzhw krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:46:12.130270 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:12.120357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43572 (local address 127.0.98.62:43293)
0927 10:46:12.120622 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:12.120627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:12.120892 (+ 265us) server_negotiation.cc:407] Connection header received
0927 10:46:12.121042 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:12.121045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:12.121094 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:12.121192 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:12.122044 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:12.122841 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:12.123939 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:12.124283 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:12.124405 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:12.126043 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:12.126048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:12.126052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:12.126092 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:12.128184 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:12.128662 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:12.128666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:12.128668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:12.128759 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:12.129325 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:12.129328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:12.129330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:12.129553 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:12.129768 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:12.129953 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 10:46:12.130083 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":91,"threads_started":1}
I20260927 10:46:12.130573 13354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:12.116832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34625 (local address 127.0.98.62:43293)
0927 10:46:12.117001 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:12.117006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:12.117989 (+ 983us) server_negotiation.cc:407] Connection header received
0927 10:46:12.119017 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:12.119021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:12.119082 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:12.119197 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:12.120659 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:12.121422 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:12.122862 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:12.123179 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:12.123291 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:12.126294 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:12.126298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:12.126300 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:12.126351 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:12.128123 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:12.128739 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:12.128743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:12.128745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:12.128869 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:12.129293 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:12.129296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:12.129298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:12.129475 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:12.129589 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:12.130243 (+ 654us) server_negotiation.cc:299] Negotiation successful
0927 10:46:12.130367 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 10:46:12.132563 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:12.117294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43293 (local address 127.0.98.3:34625)
0927 10:46:12.117794 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:12.117850 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:12.118790 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:12.119436 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:12.119447 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:12.119806 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:12.120488 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:12.120509 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:12.121569 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:12.121575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:12.122737 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:12.122747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:12.123368 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:12.123373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:12.123594 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:12.124294 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:12.124325 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:12.126132 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:12.128247 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:12.128255 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:12.128275 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:12.128661 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:12.128980 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:12.128984 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:12.128986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:12.129189 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:12.129588 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:12.129595 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:12.129946 (+ 351us) client_negotiation.cc:769] Sending connection context
0927 10:46:12.130740 (+ 794us) client_negotiation.cc:240] Negotiation successful
0927 10:46:12.131154 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":153,"threads_started":1}
I20260927 10:46:12.133836 13620 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43293
I20260927 10:46:12.134140 13620 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:12.134903 13620 heartbeater.cc:511] Master 127.0.98.62:43293 requested a full tablet report, sending...
W20260927 10:46:12.136590 13620 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43293 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 10:46:12.136751 13620 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:12.136938 13620 heartbeater.cc:511] Master 127.0.98.62:43293 requested a full tablet report, sending...
I20260927 10:46:12.138290 13620 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:12.138502 13620 heartbeater.cc:511] Master 127.0.98.62:43293 requested a full tablet report, sending...
W20260927 10:46:12.139732 13620 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 10:46:12 dist-test-slave-dzhw krb5kdc[12158](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790505972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:12.595659 13352 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:12.595986 13352 heartbeater.cc:511] Master 127.0.98.62:43293 requested a full tablet report, sending...
I20260927 10:46:12.597187 13151 ts_manager.cc:194] Registered new tserver with Master: 0a860b4df99b48afa8a3634737a191aa (127.0.98.1:34397)
I20260927 10:46:12.598937 13151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50251
Sep 27 10:46:12 dist-test-slave-dzhw krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 10:46:12.871284 13486 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:12.871629 13486 heartbeater.cc:511] Master 127.0.98.62:43293 requested a full tablet report, sending...
I20260927 10:46:12.872759 13151 ts_manager.cc:194] Registered new tserver with Master: fcc6bbd127b24986958f2053b6fc8cb7 (127.0.98.2:44915)
I20260927 10:46:12.873586 13151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43471
Sep 27 10:46:13 dist-test-slave-dzhw krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 10:46:13.140019 13620 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:13.140321 13620 heartbeater.cc:511] Master 127.0.98.62:43293 requested a full tablet report, sending...
I20260927 10:46:13.141170 13151 ts_manager.cc:194] Registered new tserver with Master: 491994d1f4d84b31b1515a52c3fd084d (127.0.98.3:36081)
I20260927 10:46:13.141949 13151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34625
I20260927 10:46:13.145339 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:46:13.158279 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:13.147588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43584 (local address 127.0.98.62:43293)
0927 10:46:13.148343 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:13.148347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:13.148560 (+ 213us) server_negotiation.cc:407] Connection header received
0927 10:46:13.148721 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:13.148725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:13.148772 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:13.148881 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:13.149842 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:13.150656 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:13.151764 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:13.152079 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:13.152189 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:13.153964 (+ 1775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:13.153968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:13.153971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:13.153998 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:13.155890 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:13.156412 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:13.156415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:13.156417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:13.156506 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:13.156796 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:13.156800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:13.156802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:13.157021 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:13.157143 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:13.157942 (+ 799us) server_negotiation.cc:299] Negotiation successful
0927 10:46:13.158076 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":95,"threads_started":1}
I20260927 10:46:13.164072 13151 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:46:13.171368 13151 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:46:13.197741 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:13.184635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44915 (local address 127.0.0.1:42942)
0927 10:46:13.186030 (+ 1395us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:13.186112 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:13.186272 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:13.187462 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:13.187469 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:13.187495 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:46:13.187628 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:13.187636 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:13.189446 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:13.189450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:13.191014 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:13.191026 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:13.197132 (+ 6106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:13.197136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:13.197273 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:13.197318 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 10:46:13.197439 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 10:46:13.197580 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1204,"thread_start_us":132,"threads_started":1}
I20260927 10:46:13.198092 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:13.185493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59748 (local address 127.0.98.1:34397)
0927 10:46:13.186598 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:13.186605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:13.188888 (+ 2283us) server_negotiation.cc:407] Connection header received
0927 10:46:13.191335 (+ 2447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:13.191341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:13.191570 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:13.191742 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:46:13.192148 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:13.193218 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:13.194713 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:13.195554 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:13.195658 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:13.195839 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:13.197816 (+ 1977us) server_negotiation.cc:299] Negotiation successful
0927 10:46:13.197938 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":999,"thread_start_us":125,"threads_started":1}
I20260927 10:46:13.197744 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:13.184652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34397 (local address 127.0.0.1:59748)
0927 10:46:13.188750 (+ 4098us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:13.188767 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:13.191167 (+ 2400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:13.191909 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:13.191912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:13.191930 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:46:13.192017 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:13.192022 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:13.193354 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:13.193358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:13.194553 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:13.194565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:13.197412 (+ 2847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:13.197418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:13.197507 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:13.197526 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 10:46:13.197603 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 10:46:13.197660 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4030,"thread_start_us":116,"threads_started":1}
I20260927 10:46:13.199020 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:13.185501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42942 (local address 127.0.98.2:44915)
0927 10:46:13.186778 (+ 1277us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:13.186784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:13.186810 (+ 26us) server_negotiation.cc:407] Connection header received
0927 10:46:13.186890 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:13.186895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:13.187113 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:13.187270 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:46:13.187778 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:13.189279 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:13.195927 (+ 6648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:13.196788 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:13.196904 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:13.197054 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:13.198756 (+ 1702us) server_negotiation.cc:299] Negotiation successful
0927 10:46:13.198861 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1185,"thread_start_us":125,"threads_started":1}
I20260927 10:46:13.202390 13286 tablet_service.cc:1467] Processing CreateTablet for tablet 6e917ed345c548a281b243a28b5ba8cc (DEFAULT_TABLE table=test-table [id=238ede713f6d41d2b180a8dad01b3b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:46:13.203622 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e917ed345c548a281b243a28b5ba8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:46:13.207010 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:13.186282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36081 (local address 127.0.0.1:39202)
0927 10:46:13.196786 (+ 10504us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:13.196803 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:13.196893 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:13.199203 (+ 2310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:13.199209 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:13.199229 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:46:13.199770 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:13.199781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:13.203447 (+ 3666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:13.203451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:13.204630 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:13.204642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:13.206495 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:13.206499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:13.206693 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:13.206720 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:46:13.206787 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 10:46:13.206860 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10424,"thread_start_us":84,"threads_started":1}
I20260927 10:46:13.207829 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:13.186695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39202 (local address 127.0.98.3:36081)
0927 10:46:13.198496 (+ 11801us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:13.198503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:13.198530 (+ 27us) server_negotiation.cc:407] Connection header received
0927 10:46:13.198633 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:13.198641 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:13.198871 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:13.199023 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:46:13.199913 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:13.200979 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:13.205305 (+ 4326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:13.206138 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:13.206244 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:13.206401 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:13.207564 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0927 10:46:13.207687 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11658,"thread_start_us":84,"threads_started":1}
I20260927 10:46:13.211009 13544 tablet_service.cc:1467] Processing CreateTablet for tablet 6e917ed345c548a281b243a28b5ba8cc (DEFAULT_TABLE table=test-table [id=238ede713f6d41d2b180a8dad01b3b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:46:13.212112 13544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e917ed345c548a281b243a28b5ba8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:46:13.217439 13420 tablet_service.cc:1467] Processing CreateTablet for tablet 6e917ed345c548a281b243a28b5ba8cc (DEFAULT_TABLE table=test-table [id=238ede713f6d41d2b180a8dad01b3b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:46:13.218626 13420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e917ed345c548a281b243a28b5ba8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:46:13.222002 13655 tablet_bootstrap.cc:492] T 6e917ed345c548a281b243a28b5ba8cc P 0a860b4df99b48afa8a3634737a191aa: Bootstrap starting.
I20260927 10:46:13.224417 13655 tablet_bootstrap.cc:654] T 6e917ed345c548a281b243a28b5ba8cc P 0a860b4df99b48afa8a3634737a191aa: Neither blocks nor log segments found. Creating new log.
I20260927 10:46:13.225350 13655 log.cc:824] T 6e917ed345c548a281b243a28b5ba8cc P 0a860b4df99b48afa8a3634737a191aa: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:13.226154 13656 tablet_bootstrap.cc:492] T 6e917ed345c548a281b243a28b5ba8cc P 491994d1f4d84b31b1515a52c3fd084d: Bootstrap starting.
I20260927 10:46:13.228467 13656 tablet_bootstrap.cc:654] T 6e917ed345c548a281b243a28b5ba8cc P 491994d1f4d84b31b1515a52c3fd084d: Neither blocks nor log segments found. Creating new log.
I20260927 10:46:13.229393 13656 log.cc:824] T 6e917ed345c548a281b243a28b5ba8cc P 491994d1f4d84b31b1515a52c3fd084d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:13.231738 13655 tablet_bootstrap.cc:492] T 6e917ed345c548a281b243a28b5ba8cc P 0a860b4df99b48afa8a3634737a191aa: No bootstrap required, opened a new log
I20260927 10:46:13.232003 13655 ts_tablet_manager.cc:1397] T 6e917ed345c548a281b243a28b5ba8cc P 0a860b4df99b48afa8a3634737a191aa: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260927 10:46:13.234783 13658 tablet_bootstrap.cc:492] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7: Bootstrap starting.
I20260927 10:46:13.236372 13655 raft_consensus.cc:348] T 6e917ed345c548a281b243a28b5ba8cc P 0a860b4df99b48afa8a3634737a191aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcc6bbd127b24986958f2053b6fc8cb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44915 } } peers { permanent_uuid: "0a860b4df99b48afa8a3634737a191aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34397 } } peers { permanent_uuid: "491994d1f4d84b31b1515a52c3fd084d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 10:46:13.236786 13655 raft_consensus.cc:374] T 6e917ed345c548a281b243a28b5ba8cc P 0a860b4df99b48afa8a3634737a191aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:46:13.236882 13655 raft_consensus.cc:729] T 6e917ed345c548a281b243a28b5ba8cc P 0a860b4df99b48afa8a3634737a191aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a860b4df99b48afa8a3634737a191aa, State: Initialized, Role: FOLLOWER
I20260927 10:46:13.237182 13658 tablet_bootstrap.cc:654] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7: Neither blocks nor log segments found. Creating new log.
I20260927 10:46:13.237556 13655 consensus_queue.cc:260] T 6e917ed345c548a281b243a28b5ba8cc P 0a860b4df99b48afa8a3634737a191aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcc6bbd127b24986958f2053b6fc8cb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44915 } } peers { permanent_uuid: "0a860b4df99b48afa8a3634737a191aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34397 } } peers { permanent_uuid: "491994d1f4d84b31b1515a52c3fd084d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 10:46:13.238052 13658 log.cc:824] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:13.238781 13655 ts_tablet_manager.cc:1428] T 6e917ed345c548a281b243a28b5ba8cc P 0a860b4df99b48afa8a3634737a191aa: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 10:46:13.239189 13352 heartbeater.cc:503] Master 127.0.98.62:43293 was elected leader, sending a full tablet report...
I20260927 10:46:13.242369 13658 tablet_bootstrap.cc:492] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7: No bootstrap required, opened a new log
I20260927 10:46:13.242681 13658 ts_tablet_manager.cc:1397] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260927 10:46:13.245481 13656 tablet_bootstrap.cc:492] T 6e917ed345c548a281b243a28b5ba8cc P 491994d1f4d84b31b1515a52c3fd084d: No bootstrap required, opened a new log
I20260927 10:46:13.245751 13656 ts_tablet_manager.cc:1397] T 6e917ed345c548a281b243a28b5ba8cc P 491994d1f4d84b31b1515a52c3fd084d: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20260927 10:46:13.247030 13658 raft_consensus.cc:348] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcc6bbd127b24986958f2053b6fc8cb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44915 } } peers { permanent_uuid: "0a860b4df99b48afa8a3634737a191aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34397 } } peers { permanent_uuid: "491994d1f4d84b31b1515a52c3fd084d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 10:46:13.247390 13658 raft_consensus.cc:374] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:46:13.247484 13658 raft_consensus.cc:729] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcc6bbd127b24986958f2053b6fc8cb7, State: Initialized, Role: FOLLOWER
I20260927 10:46:13.248045 13658 consensus_queue.cc:260] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcc6bbd127b24986958f2053b6fc8cb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44915 } } peers { permanent_uuid: "0a860b4df99b48afa8a3634737a191aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34397 } } peers { permanent_uuid: "491994d1f4d84b31b1515a52c3fd084d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 10:46:13.249464 13658 ts_tablet_manager.cc:1428] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260927 10:46:13.249861 13486 heartbeater.cc:503] Master 127.0.98.62:43293 was elected leader, sending a full tablet report...
I20260927 10:46:13.250480 13656 raft_consensus.cc:348] T 6e917ed345c548a281b243a28b5ba8cc P 491994d1f4d84b31b1515a52c3fd084d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcc6bbd127b24986958f2053b6fc8cb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44915 } } peers { permanent_uuid: "0a860b4df99b48afa8a3634737a191aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34397 } } peers { permanent_uuid: "491994d1f4d84b31b1515a52c3fd084d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 10:46:13.250766 13656 raft_consensus.cc:374] T 6e917ed345c548a281b243a28b5ba8cc P 491994d1f4d84b31b1515a52c3fd084d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:46:13.250824 13656 raft_consensus.cc:729] T 6e917ed345c548a281b243a28b5ba8cc P 491994d1f4d84b31b1515a52c3fd084d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491994d1f4d84b31b1515a52c3fd084d, State: Initialized, Role: FOLLOWER
I20260927 10:46:13.251308 13656 consensus_queue.cc:260] T 6e917ed345c548a281b243a28b5ba8cc P 491994d1f4d84b31b1515a52c3fd084d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcc6bbd127b24986958f2053b6fc8cb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44915 } } peers { permanent_uuid: "0a860b4df99b48afa8a3634737a191aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34397 } } peers { permanent_uuid: "491994d1f4d84b31b1515a52c3fd084d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 10:46:13.252982 13656 ts_tablet_manager.cc:1428] T 6e917ed345c548a281b243a28b5ba8cc P 491994d1f4d84b31b1515a52c3fd084d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 10:46:13.253363 13620 heartbeater.cc:503] Master 127.0.98.62:43293 was elected leader, sending a full tablet report...
I20260927 10:46:13.254307 13663 raft_consensus.cc:482] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:46:13.254503 13663 raft_consensus.cc:504] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcc6bbd127b24986958f2053b6fc8cb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44915 } } peers { permanent_uuid: "0a860b4df99b48afa8a3634737a191aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34397 } } peers { permanent_uuid: "491994d1f4d84b31b1515a52c3fd084d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 10:46:13.257146 13663 leader_election.cc:288] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a860b4df99b48afa8a3634737a191aa (127.0.98.1:34397), 491994d1f4d84b31b1515a52c3fd084d (127.0.98.3:36081)
I20260927 10:46:13.262017 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:13.257039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41089 (local address 127.0.98.3:36081)
0927 10:46:13.257150 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:13.257156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:13.257498 (+ 342us) server_negotiation.cc:407] Connection header received
0927 10:46:13.257706 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:13.257709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:13.257754 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:13.257803 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:46:13.258318 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:13.259127 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:13.260723 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:13.261339 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:13.261414 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:13.261481 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:13.261810 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 10:46:13.261888 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 10:46:13.262727 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:13.256754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36081 (local address 127.0.98.2:41089)
0927 10:46:13.257419 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:13.257431 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:13.257588 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:13.257941 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:13.257944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:13.257968 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:46:13.258187 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:13.258193 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:13.259273 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:13.259277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:13.260572 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:13.260583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:13.261574 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:13.261579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:13.261671 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:13.261694 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:46:13.262447 (+ 753us) client_negotiation.cc:240] Negotiation successful
0927 10:46:13.262535 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":45,"spinlock_wait_cycles":45824,"thread_start_us":87,"threads_started":1}
I20260927 10:46:13.263232 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:13.256312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40043 (local address 127.0.98.1:34397)
0927 10:46:13.256436 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:13.256440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:13.257497 (+ 1057us) server_negotiation.cc:407] Connection header received
0927 10:46:13.257708 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:13.257711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:13.257773 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:13.257821 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:46:13.258318 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:13.259125 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:13.261616 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:13.262254 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:13.262331 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:13.262384 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:13.263039 (+ 655us) server_negotiation.cc:299] Negotiation successful
0927 10:46:13.263120 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:46:13.263587 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:13.255962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34397 (local address 127.0.98.2:40043)
0927 10:46:13.257403 (+ 1441us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:13.257418 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:13.257588 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:13.257942 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:13.257944 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:13.257969 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:46:13.258187 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:13.258193 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:13.259273 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:13.259277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:13.261481 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:13.261492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:13.262779 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:13.262785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:13.262894 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:13.262914 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:46:13.263378 (+ 464us) client_negotiation.cc:240] Negotiation successful
0927 10:46:13.263449 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1374,"mutex_wait_us":1070,"spinlock_wait_cycles":26624,"thread_start_us":94,"threads_started":1}
I20260927 10:46:13.263567 13556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e917ed345c548a281b243a28b5ba8cc" candidate_uuid: "fcc6bbd127b24986958f2053b6fc8cb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491994d1f4d84b31b1515a52c3fd084d" is_pre_election: true
I20260927 10:46:13.263891 13556 raft_consensus.cc:2454] T 6e917ed345c548a281b243a28b5ba8cc P 491994d1f4d84b31b1515a52c3fd084d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcc6bbd127b24986958f2053b6fc8cb7 in term 0.
I20260927 10:46:13.264245 13306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e917ed345c548a281b243a28b5ba8cc" candidate_uuid: "fcc6bbd127b24986958f2053b6fc8cb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a860b4df99b48afa8a3634737a191aa" is_pre_election: true
I20260927 10:46:13.264525 13306 raft_consensus.cc:2454] T 6e917ed345c548a281b243a28b5ba8cc P 0a860b4df99b48afa8a3634737a191aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcc6bbd127b24986958f2053b6fc8cb7 in term 0.
I20260927 10:46:13.265100 13374 leader_election.cc:302] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491994d1f4d84b31b1515a52c3fd084d, fcc6bbd127b24986958f2053b6fc8cb7; no voters:
I20260927 10:46:13.265905 13662 raft_consensus.cc:2781] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:46:13.266129 13662 raft_consensus.cc:482] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:46:13.266253 13662 raft_consensus.cc:3037] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:46:13.267757 13662 raft_consensus.cc:504] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcc6bbd127b24986958f2053b6fc8cb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44915 } } peers { permanent_uuid: "0a860b4df99b48afa8a3634737a191aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34397 } } peers { permanent_uuid: "491994d1f4d84b31b1515a52c3fd084d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 10:46:13.268518 13306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e917ed345c548a281b243a28b5ba8cc" candidate_uuid: "fcc6bbd127b24986958f2053b6fc8cb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a860b4df99b48afa8a3634737a191aa"
I20260927 10:46:13.268651 13306 raft_consensus.cc:3037] T 6e917ed345c548a281b243a28b5ba8cc P 0a860b4df99b48afa8a3634737a191aa [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:46:13.269955 13306 raft_consensus.cc:2454] T 6e917ed345c548a281b243a28b5ba8cc P 0a860b4df99b48afa8a3634737a191aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcc6bbd127b24986958f2053b6fc8cb7 in term 1.
I20260927 10:46:13.270341 13374 leader_election.cc:302] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a860b4df99b48afa8a3634737a191aa, fcc6bbd127b24986958f2053b6fc8cb7; no voters:
I20260927 10:46:13.270612 13663 raft_consensus.cc:2781] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:46:13.270844 13663 raft_consensus.cc:686] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7 [term 1 LEADER]: Becoming Leader. State: Replica: fcc6bbd127b24986958f2053b6fc8cb7, State: Running, Role: LEADER
I20260927 10:46:13.271216 13663 consensus_queue.cc:237] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcc6bbd127b24986958f2053b6fc8cb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44915 } } peers { permanent_uuid: "0a860b4df99b48afa8a3634737a191aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34397 } } peers { permanent_uuid: "491994d1f4d84b31b1515a52c3fd084d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 10:46:13.274003 13662 leader_election.cc:288] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7 [CANDIDATE]: Term 1 election: Requested vote from peers 0a860b4df99b48afa8a3634737a191aa (127.0.98.1:34397), 491994d1f4d84b31b1515a52c3fd084d (127.0.98.3:36081)
I20260927 10:46:13.274434 13556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e917ed345c548a281b243a28b5ba8cc" candidate_uuid: "fcc6bbd127b24986958f2053b6fc8cb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491994d1f4d84b31b1515a52c3fd084d"
I20260927 10:46:13.274593 13556 raft_consensus.cc:3037] T 6e917ed345c548a281b243a28b5ba8cc P 491994d1f4d84b31b1515a52c3fd084d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:46:13.274624 13151 catalog_manager.cc:5495] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to fcc6bbd127b24986958f2053b6fc8cb7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "fcc6bbd127b24986958f2053b6fc8cb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcc6bbd127b24986958f2053b6fc8cb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a860b4df99b48afa8a3634737a191aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491994d1f4d84b31b1515a52c3fd084d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } health_report { overall_health: UNKNOWN } } }
I20260927 10:46:13.275969 13556 raft_consensus.cc:2454] T 6e917ed345c548a281b243a28b5ba8cc P 491994d1f4d84b31b1515a52c3fd084d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcc6bbd127b24986958f2053b6fc8cb7 in term 1.
I20260927 10:46:13.297164 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:13.292349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42948 (local address 127.0.98.2:44915)
0927 10:46:13.292468 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:13.292472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:13.292487 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:46:13.292569 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:13.292573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:13.292640 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:13.292758 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:46:13.293036 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:13.293816 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:13.295108 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:13.295466 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:13.295573 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:13.296003 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:46:13.296590 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:46:13.296734 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:13.296970 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 10:46:13.297026 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 10:46:13.305766 13353 tablet.cc:2369] T 6e917ed345c548a281b243a28b5ba8cc P 0a860b4df99b48afa8a3634737a191aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:46:13.310501 13556 raft_consensus.cc:1261] T 6e917ed345c548a281b243a28b5ba8cc P 491994d1f4d84b31b1515a52c3fd084d [term 1 FOLLOWER]: Refusing update from remote peer fcc6bbd127b24986958f2053b6fc8cb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:46:13.311429 13667 consensus_queue.cc:1035] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491994d1f4d84b31b1515a52c3fd084d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:46:13.313213 13306 raft_consensus.cc:1261] T 6e917ed345c548a281b243a28b5ba8cc P 0a860b4df99b48afa8a3634737a191aa [term 1 FOLLOWER]: Refusing update from remote peer fcc6bbd127b24986958f2053b6fc8cb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:46:13.313939 13667 consensus_queue.cc:1035] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a860b4df99b48afa8a3634737a191aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:46:13.326282 13670 mvcc.cc:205] Tried to move back new op lower bound from 7333912466670501888 to 7333912466521886720. Current Snapshot: MvccSnapshot[applied={T|T < 7333912466670501888}]
I20260927 10:46:13.326478 13673 mvcc.cc:205] Tried to move back new op lower bound from 7333912466670501888 to 7333912466521886720. Current Snapshot: MvccSnapshot[applied={T|T < 7333912466670501888}]
I20260927 10:46:13.328860 13671 mvcc.cc:205] Tried to move back new op lower bound from 7333912466670501888 to 7333912466521886720. Current Snapshot: MvccSnapshot[applied={T|T < 7333912466670501888}]
I20260927 10:46:13.337313 13154 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:46:13.337508 13154 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:46:13.340561 13154 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba620296e76d440ab0f90e22bc347339: Sending DeleteTablet for 3 replicas of tablet 6e917ed345c548a281b243a28b5ba8cc
I20260927 10:46:13.341903 13286 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e917ed345c548a281b243a28b5ba8cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:46:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59748
I20260927 10:46:13.342505 13544 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e917ed345c548a281b243a28b5ba8cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:46:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39202
I20260927 10:46:13.343741 13420 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e917ed345c548a281b243a28b5ba8cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:46:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42942
I20260927 10:46:13.344489 13679 tablet_replica.cc:330] T 6e917ed345c548a281b243a28b5ba8cc P 0a860b4df99b48afa8a3634737a191aa: stopping tablet replica
I20260927 10:46:13.344722 13680 tablet_replica.cc:330] T 6e917ed345c548a281b243a28b5ba8cc P 491994d1f4d84b31b1515a52c3fd084d: stopping tablet replica
I20260927 10:46:13.344856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 13192
I20260927 10:46:13.346335 13681 tablet_replica.cc:330] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7: stopping tablet replica
I20260927 10:46:13.346625 13681 raft_consensus.cc:2229] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:46:13.347025 13681 raft_consensus.cc:2258] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:46:13.345106 13680 raft_consensus.cc:2229] T 6e917ed345c548a281b243a28b5ba8cc P 491994d1f4d84b31b1515a52c3fd084d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:46:13.347553 13680 raft_consensus.cc:2258] T 6e917ed345c548a281b243a28b5ba8cc P 491994d1f4d84b31b1515a52c3fd084d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:46:13.349076 13680 ts_tablet_manager.cc:1905] T 6e917ed345c548a281b243a28b5ba8cc P 491994d1f4d84b31b1515a52c3fd084d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:46:13.350286 13681 ts_tablet_manager.cc:1905] T 6e917ed345c548a281b243a28b5ba8cc P fcc6bbd127b24986958f2053b6fc8cb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:46:13.353124 13680 ts_tablet_manager.cc:1918] T 6e917ed345c548a281b243a28b5ba8cc P 491994d1f4d84b31b1515a52c3fd084d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260927 10:46:13.354146 13139 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34397 (error 108)
I20260927 10:46:13.354373 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 13356
I20260927 10:46:13.355966 13680 log.cc:1196] T 6e917ed345c548a281b243a28b5ba8cc P 491994d1f4d84b31b1515a52c3fd084d: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/6e917ed345c548a281b243a28b5ba8cc
I20260927 10:46:13.356376 13680 ts_tablet_manager.cc:1939] T 6e917ed345c548a281b243a28b5ba8cc P 491994d1f4d84b31b1515a52c3fd084d: Deleting consensus metadata
I20260927 10:46:13.356767 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:13.356409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34397 (local address 127.0.0.1:59750)
0927 10:46:13.356609 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:13.356678 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260927 10:46:13.357069 13139 catalog_manager.cc:4584] TS 0a860b4df99b48afa8a3634737a191aa (127.0.98.1:34397): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6e917ed345c548a281b243a28b5ba8cc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34397: connect: Connection refused (error 111)
I20260927 10:46:13.357816 13139 catalog_manager.cc:4841] TS 491994d1f4d84b31b1515a52c3fd084d (127.0.98.3:36081): tablet 6e917ed345c548a281b243a28b5ba8cc (table test-table [id=238ede713f6d41d2b180a8dad01b3b2a]) successfully deleted
W20260927 10:46:13.363711 13649 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:46:13.363131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44915 (local address 127.0.0.1:42950)
0927 10:46:13.363299 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:13.363314 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:13.363403 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:13.363591 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 10:46:13.364400 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 13489
I20260927 10:46:13.372125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 13121
I20260927 10:46:13.379621 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 10:46:13.724644 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 10:46:13.724764 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 10:46:14.070098 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 10:46:14.070 UTC [12188] LOG: received smart shutdown request
2026-09-27 10:46:14.074 UTC [12188] LOG: background worker "logical replication launcher" (PID 12197) exited with exit code 1
2026-09-27 10:46:14.074 UTC [12192] LOG: shutting down
2026-09-27 10:46:14.255 UTC [12188] LOG: database system is shut down
2026-09-27T10:46:14Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49618 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:46:14 dist-test-slave-dzhw krb5kdc[13693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:46:14 dist-test-slave-dzhw krb5kdc[13693](info): set up 2 sockets
Sep 27 10:46:14 dist-test-slave-dzhw krb5kdc[13693](info): commencing operation
krb5kdc: starting...
W20260927 10:46:15.418614 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:46:15 dist-test-slave-dzhw krb5kdc[13693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:46:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:46:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:46:15.537685 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42999
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42999,127.0.98.61:35013,127.0.98.60:44805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:46:15.646903 13706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:15.647179 13706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:15.647220 13706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:15.651222 13706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:46:15.651283 13706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:15.651302 13706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:15.651319 13706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:46:15.651333 13706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:46:15.656877 13706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42999,127.0.98.61:35013,127.0.98.60:44805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:15.657922 13706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:15.658906 13706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:15.667099 13714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:15.667156 13711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:15.667160 13712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:15.667259 13706 server_base.cc:1034] running on GCE node
I20260927 10:46:15.667858 13706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:15.668938 13706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:15.670115 13706 hybrid_clock.cc:648] HybridClock initialized: now 1790505975670108 us; error 30 us; skew 500 ppm
Sep 27 10:46:15 dist-test-slave-dzhw krb5kdc[13693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:15.672933 13706 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:46:15.673804 13706 webserver.cc:460] Webserver started at http://127.0.98.62:40129/ using document root <none> and password file <none>
I20260927 10:46:15.674307 13706 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:15.674363 13706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:15.674575 13706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:46:15.676528 13706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "89c6a65a734c4fb98df3270a485e8643"
format_stamp: "Formatted at 2026-09-27 10:46:15 on dist-test-slave-dzhw"
server_key: "82561b40bb1a1943d7e1863d0a032091"
server_key_iv: "8c2c737d5020dfd16d7f658da8b5c2b5"
server_key_version: "encryptionkey@0"
I20260927 10:46:15.677016 13706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "89c6a65a734c4fb98df3270a485e8643"
format_stamp: "Formatted at 2026-09-27 10:46:15 on dist-test-slave-dzhw"
server_key: "82561b40bb1a1943d7e1863d0a032091"
server_key_iv: "8c2c737d5020dfd16d7f658da8b5c2b5"
server_key_version: "encryptionkey@0"
I20260927 10:46:15.680737 13706 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:46:15.683079 13721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:15.684324 13706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:46:15.684422 13706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "89c6a65a734c4fb98df3270a485e8643"
format_stamp: "Formatted at 2026-09-27 10:46:15 on dist-test-slave-dzhw"
server_key: "82561b40bb1a1943d7e1863d0a032091"
server_key_iv: "8c2c737d5020dfd16d7f658da8b5c2b5"
server_key_version: "encryptionkey@0"
I20260927 10:46:15.684510 13706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:15.709088 13706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:15.712069 13706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:15.712234 13706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:15.719029 13706 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42999
I20260927 10:46:15.719049 13773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42999 every 8 connection(s)
I20260927 10:46:15.719952 13706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:46:15.722690 13774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:46:15.723898 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 13706
I20260927 10:46:15.724040 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:46:15.724326 392 external_mini_cluster.cc:1428] Setting key a87c316a91303369fdcbac1720290abb
I20260927 10:46:15.727010 13774 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 10:46:15 dist-test-slave-dzhw krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:46:15.743184 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:15.729775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42999 (local address 127.0.0.1:47522)
0927 10:46:15.730285 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:15.730338 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:15.731730 (+ 1392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:15.732472 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:15.732482 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:15.732962 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:15.733342 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:15.733365 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:15.734587 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:15.734590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:15.735365 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:15.735375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:15.736013 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:15.736016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:15.736113 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:15.736739 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:15.736764 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:15.738470 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:15.740314 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:15.740322 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:15.740332 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:15.740548 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:15.740859 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:15.740862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:15.740864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:15.740968 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:15.741527 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:15.741533 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:15.741666 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 10:46:15.741828 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 10:46:15.742180 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":102,"thread_start_us":103,"threads_started":1}
I20260927 10:46:15.743193 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:15.729775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47522 (local address 127.0.98.62:42999)
0927 10:46:15.730261 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:15.730279 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:15.730480 (+ 201us) server_negotiation.cc:407] Connection header received
0927 10:46:15.732067 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:15.732071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:15.732259 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:15.732411 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:15.733503 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:15.734474 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:15.735467 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:15.735845 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:15.735991 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:15.738607 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:15.738612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:15.738622 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:15.738642 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:15.740180 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:15.740660 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:15.740670 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:15.740671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:15.740765 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:15.741057 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:15.741060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:15.741061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:15.741420 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:15.741553 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:15.741894 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 10:46:15.742130 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":351,"spinlock_wait_cycles":16128,"thread_start_us":201,"threads_started":1}
I20260927 10:46:15.745425 13774 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35013 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:46:15.746191 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:15.745875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35013 (local address 127.0.0.1:50624)
0927 10:46:15.746018 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:15.746123 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 10:46:15.746366 13723 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35013: connect: Connection refused (error 111)
I20260927 10:46:15.747143 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:15.746959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35013 (local address 127.0.0.1:50630)
0927 10:46:15.747048 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:15.747093 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 10:46:15.747277 13774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35013: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35013: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 10:46:15.753475 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35013
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42999,127.0.98.61:35013,127.0.98.60:44805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:46:15.796597 13774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35013 } attempt: 1
I20260927 10:46:15.797646 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:15.797287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35013 (local address 127.0.0.1:50638)
0927 10:46:15.797492 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:15.797570 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 10:46:15.798395 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:15.798172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35013 (local address 127.0.0.1:50648)
0927 10:46:15.798295 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:15.798343 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 10:46:15.798583 13774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35013: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35013: connect: Connection refused (error 111)
I20260927 10:46:15.866781 13774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35013 } attempt: 2
W20260927 10:46:15.867075 13781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:15.867443 13781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:15.867573 13781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 10:46:15.867741 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:15.867377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35013 (local address 127.0.0.1:50650)
0927 10:46:15.867572 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:15.867657 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 10:46:15.868577 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:15.868282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35013 (local address 127.0.0.1:50652)
0927 10:46:15.868434 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:15.868501 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 10:46:15.868722 13774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35013: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35013: connect: Connection refused (error 111)
W20260927 10:46:15.872488 13781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:46:15.872576 13781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:15.872620 13781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:15.872658 13781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:46:15.872694 13781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:46:15.878510 13781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42999,127.0.98.61:35013,127.0.98.60:44805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:15.880313 13781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:15.881793 13781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:15.890759 13787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:15.890759 13789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:15.890813 13786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:15.891120 13781 server_base.cc:1034] running on GCE node
I20260927 10:46:15.891700 13781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:15.892838 13781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:15.894021 13781 hybrid_clock.cc:648] HybridClock initialized: now 1790505975893997 us; error 60 us; skew 500 ppm
Sep 27 10:46:15 dist-test-slave-dzhw krb5kdc[13693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:15.896808 13781 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 10:46:15.897635 13781 webserver.cc:460] Webserver started at http://127.0.98.61:37805/ using document root <none> and password file <none>
I20260927 10:46:15.898149 13781 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:15.898213 13781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:15.898375 13781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:46:15.900502 13781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-1/data/instance:
uuid: "26cc089c608342e28407fd9bf859a85e"
format_stamp: "Formatted at 2026-09-27 10:46:15 on dist-test-slave-dzhw"
server_key: "07c300faa3d090ac0b5dbecac9c88570"
server_key_iv: "2dc5a6624475416b6eafeed7e28328cb"
server_key_version: "encryptionkey@0"
I20260927 10:46:15.901021 13781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-1/wal/instance:
uuid: "26cc089c608342e28407fd9bf859a85e"
format_stamp: "Formatted at 2026-09-27 10:46:15 on dist-test-slave-dzhw"
server_key: "07c300faa3d090ac0b5dbecac9c88570"
server_key_iv: "2dc5a6624475416b6eafeed7e28328cb"
server_key_version: "encryptionkey@0"
I20260927 10:46:15.904666 13781 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 10:46:15.907405 13796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:15.909008 13781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:46:15.909149 13781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-1/wal
uuid: "26cc089c608342e28407fd9bf859a85e"
format_stamp: "Formatted at 2026-09-27 10:46:15 on dist-test-slave-dzhw"
server_key: "07c300faa3d090ac0b5dbecac9c88570"
server_key_iv: "2dc5a6624475416b6eafeed7e28328cb"
server_key_version: "encryptionkey@0"
I20260927 10:46:15.909250 13781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:15.932654 13781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:15.936898 13781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:15.937059 13781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:15.943995 13781 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35013
I20260927 10:46:15.944027 13848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35013 every 8 connection(s)
I20260927 10:46:15.944941 13781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-1/data/info.pb
I20260927 10:46:15.947659 13849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:46:15.950699 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 13781
I20260927 10:46:15.950840 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-1/wal/instance
I20260927 10:46:15.951125 392 external_mini_cluster.cc:1428] Setting key 2de92ad089faba86217794e0e3e2af5a
I20260927 10:46:15.951500 13849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:46:15.959944 13774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35013 } attempt: 3
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 10:46:15 dist-test-slave-dzhw krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 10:46:15 dist-test-slave-dzhw krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 10:46:15.968905 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:15.953630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47526 (local address 127.0.98.62:42999)
0927 10:46:15.953778 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:15.953782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:15.954647 (+ 865us) server_negotiation.cc:407] Connection header received
0927 10:46:15.955591 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:15.955595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:15.955655 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:15.955758 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:15.957224 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:15.957751 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:15.959537 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:15.959797 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:15.959892 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:15.963907 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:15.963911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:15.963913 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:15.963934 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:15.965845 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:15.966360 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:15.966365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:15.966367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:15.966475 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:15.966818 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:15.966821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:15.966823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:15.967001 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:15.967100 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:15.968606 (+ 1506us) server_negotiation.cc:299] Negotiation successful
0927 10:46:15.968748 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:46:15.969447 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:15.953989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42999 (local address 127.0.0.1:47526)
0927 10:46:15.954492 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:15.954534 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:15.955393 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:15.955922 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:15.955929 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:15.956293 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:15.957029 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:15.957052 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:15.957888 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:15.957895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:15.959389 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:15.959404 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:15.959986 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:15.959989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:15.960214 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:15.960949 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:15.960966 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:15.963767 (+ 2801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:15.965958 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:15.965964 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:15.965985 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:15.966254 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:15.966570 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:15.966574 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:15.966576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:15.966716 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:15.967101 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:15.967108 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:15.967439 (+ 331us) client_negotiation.cc:769] Sending connection context
0927 10:46:15.967659 (+ 220us) client_negotiation.cc:240] Negotiation successful
0927 10:46:15.968019 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":117,"threads_started":1}
I20260927 10:46:15.970899 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:15.960400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35013 (local address 127.0.0.1:50666)
0927 10:46:15.960527 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:15.960545 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:15.960620 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:15.961299 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:15.961302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:15.961482 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:15.961573 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:15.961578 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:15.962701 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:15.962708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:15.963552 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:15.963564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:15.964437 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:15.964443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:15.964535 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:15.965025 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:15.965036 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:15.966756 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:15.969186 (+ 2430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:15.969190 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:15.969192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:15.969422 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:15.969772 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:15.969775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:15.969777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:15.969872 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:15.970375 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:15.970378 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:15.970498 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 10:46:15.970619 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 10:46:15.970757 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 10:46:15.971181 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:15.960573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50666 (local address 127.0.98.61:35013)
0927 10:46:15.960858 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:15.960869 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:15.960888 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:46:15.960921 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:15.960925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:15.961153 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:15.961316 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:15.961689 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:15.962554 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:15.963671 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:15.964280 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:15.964722 (+ 442us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:15.966940 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:15.966952 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:15.966971 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:15.967002 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:15.969072 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:15.969544 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:15.969552 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:15.969555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:15.969669 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:15.969983 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:15.969989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:15.969991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:15.970284 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:15.970440 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:15.970733 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 10:46:15.970886 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":109,"threads_started":1}
I20260927 10:46:15.971407 13849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35013 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:46:15.972898 13774 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:46:15.973544 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:15.973337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44805 (local address 127.0.0.1:58644)
0927 10:46:15.973434 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:15.973479 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 10:46:15.974149 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:15.973931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44805 (local address 127.0.0.1:58654)
0927 10:46:15.974056 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:15.974099 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 10:46:15.974510 13774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44805: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44805: connect: Connection refused (error 111)
Sep 27 10:46:15 dist-test-slave-dzhw krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 10:46:15.981213 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:15.971912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35013 (local address 127.0.0.1:50678)
0927 10:46:15.972072 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:15.972089 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:15.972163 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:15.972519 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:15.972521 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:15.972692 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:15.972848 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:15.972856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:15.973770 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:15.973773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:15.974438 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:15.974448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:15.974943 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:15.974946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:15.975035 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:15.975524 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:15.975537 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:15.976896 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:15.979575 (+ 2679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:15.979578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:15.979581 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:15.979831 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:15.980090 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:15.980094 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:15.980096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:15.980175 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:15.980643 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:15.980646 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:15.980734 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 10:46:15.980854 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 10:46:15.981018 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 10:46:15.981221 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:15.971941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50678 (local address 127.0.98.61:35013)
0927 10:46:15.972354 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:15.972360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:15.972368 (+ 8us) server_negotiation.cc:407] Connection header received
0927 10:46:15.972389 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:15.972391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:15.972419 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:15.972510 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:15.972937 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:15.973667 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:15.974619 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:15.974838 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:15.974947 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:15.977045 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:15.977049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:15.977051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:15.977079 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:15.979378 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:15.979923 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:15.979929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:15.979931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:15.980005 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:15.980293 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:15.980296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:15.980297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:15.980542 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:15.980647 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:15.980924 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 10:46:15.981075 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":369,"thread_start_us":153,"threads_started":1}
I20260927 10:46:15.981478 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44805
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42999,127.0.98.61:35013,127.0.98.60:44805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:46:15.982038 13849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:46:15.982828 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:15.982499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44805 (local address 127.0.0.1:58662)
0927 10:46:15.982629 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:15.982744 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 10:46:15.983047 13797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44805: connect: Connection refused (error 111)
I20260927 10:46:15.983860 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:15.983671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44805 (local address 127.0.0.1:58678)
0927 10:46:15.983766 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:15.983814 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 10:46:15.984007 13849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44805: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44805: connect: Connection refused (error 111)
I20260927 10:46:16.005697 13774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } attempt: 1
I20260927 10:46:16.006611 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:16.006270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44805 (local address 127.0.0.1:58686)
0927 10:46:16.006451 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:16.006514 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 10:46:16.007302 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:16.007107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44805 (local address 127.0.0.1:58698)
0927 10:46:16.007207 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:16.007254 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 10:46:16.007454 13774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44805: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44805: connect: Connection refused (error 111)
I20260927 10:46:16.033262 13849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } attempt: 1
I20260927 10:46:16.034185 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:16.033877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44805 (local address 127.0.0.1:58706)
0927 10:46:16.034027 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:16.034089 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 10:46:16.034907 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:16.034721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44805 (local address 127.0.0.1:58708)
0927 10:46:16.034812 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:16.034856 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 10:46:16.035066 13849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44805: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44805: connect: Connection refused (error 111)
I20260927 10:46:16.082659 13774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } attempt: 2
I20260927 10:46:16.083455 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:16.083181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44805 (local address 127.0.0.1:58710)
0927 10:46:16.083316 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:16.083379 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 10:46:16.084173 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:16.083957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44805 (local address 127.0.0.1:58726)
0927 10:46:16.084057 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:16.084107 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 10:46:16.084336 13774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44805: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44805: connect: Connection refused (error 111)
W20260927 10:46:16.093706 13856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:16.093986 13856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:16.094027 13856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:16.098174 13856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:46:16.098237 13856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:16.098256 13856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:16.098273 13856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:46:16.098289 13856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:46:16.103216 13849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } attempt: 2
I20260927 10:46:16.103940 13856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42999,127.0.98.61:35013,127.0.98.60:44805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:16.104094 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:16.103775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44805 (local address 127.0.0.1:58742)
0927 10:46:16.103922 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:16.103984 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 10:46:16.104839 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:16.104642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44805 (local address 127.0.0.1:58756)
0927 10:46:16.104743 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:16.104788 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 10:46:16.105029 13849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44805: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44805: connect: Connection refused (error 111)
I20260927 10:46:16.105284 13856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:16.106353 13856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:16.114902 13863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:16.114905 13865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:16.114929 13862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:16.115268 13856 server_base.cc:1034] running on GCE node
I20260927 10:46:16.115754 13856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:16.116909 13856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:16.118103 13856 hybrid_clock.cc:648] HybridClock initialized: now 1790505976118080 us; error 35 us; skew 500 ppm
Sep 27 10:46:16 dist-test-slave-dzhw krb5kdc[13693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:16.121021 13856 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 10:46:16.121951 13856 webserver.cc:460] Webserver started at http://127.0.98.60:33903/ using document root <none> and password file <none>
I20260927 10:46:16.122576 13856 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:16.122666 13856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:16.122897 13856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:46:16.124949 13856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-2/data/instance:
uuid: "8f802173746b4ecbafb9b6b54e6d4777"
format_stamp: "Formatted at 2026-09-27 10:46:16 on dist-test-slave-dzhw"
server_key: "8c95df9050639462dbb0cb3d5dedea30"
server_key_iv: "af06672b52f2e8074c3c9e5c5be0bba8"
server_key_version: "encryptionkey@0"
I20260927 10:46:16.125519 13856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-2/wal/instance:
uuid: "8f802173746b4ecbafb9b6b54e6d4777"
format_stamp: "Formatted at 2026-09-27 10:46:16 on dist-test-slave-dzhw"
server_key: "8c95df9050639462dbb0cb3d5dedea30"
server_key_iv: "af06672b52f2e8074c3c9e5c5be0bba8"
server_key_version: "encryptionkey@0"
I20260927 10:46:16.129338 13856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 10:46:16.132388 13872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:16.133677 13856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:46:16.133814 13856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-2/wal
uuid: "8f802173746b4ecbafb9b6b54e6d4777"
format_stamp: "Formatted at 2026-09-27 10:46:16 on dist-test-slave-dzhw"
server_key: "8c95df9050639462dbb0cb3d5dedea30"
server_key_iv: "af06672b52f2e8074c3c9e5c5be0bba8"
server_key_version: "encryptionkey@0"
I20260927 10:46:16.133944 13856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:16.148128 13856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:16.151242 13856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:16.151433 13856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:16.158491 13856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44805
I20260927 10:46:16.158533 13924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44805 every 8 connection(s)
I20260927 10:46:16.159466 13856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-2/data/info.pb
I20260927 10:46:16.162322 13925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:46:16.166159 13925 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:46:16.167938 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 13856
I20260927 10:46:16.168087 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505812134190-392-0/minicluster-data/master-2/wal/instance
I20260927 10:46:16.168429 392 external_mini_cluster.cc:1428] Setting key a6bff5ba7a49be48f19ae11777c7c01a
Sep 27 10:46:16 dist-test-slave-dzhw krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 10:46:16 dist-test-slave-dzhw krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 10:46:16.183274 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:16.168174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47542 (local address 127.0.98.62:42999)
0927 10:46:16.168329 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:16.168333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:16.169457 (+ 1124us) server_negotiation.cc:407] Connection header received
0927 10:46:16.170529 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:16.170538 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:16.170595 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:16.170707 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:16.171822 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.172594 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.174296 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.174624 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.174759 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:16.177626 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:16.177633 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:16.177640 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:16.177674 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:16.179772 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:16.180491 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:16.180495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:16.180497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:16.180580 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:16.181016 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:16.181021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:16.181024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:16.181353 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:16.181467 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:16.182188 (+ 721us) server_negotiation.cc:299] Negotiation successful
0927 10:46:16.183094 (+ 906us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:46:16.183499 13774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } attempt: 3
I20260927 10:46:16.184074 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:16.168610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42999 (local address 127.0.0.1:47542)
0927 10:46:16.169296 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:16.169351 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:16.170297 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:16.170810 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:16.170816 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:16.171285 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:16.171644 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:16.171678 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.172827 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.172832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:16.174164 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:16.174176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.174830 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.174835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:16.174999 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:16.175635 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:16.175658 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:16.177420 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:16.179919 (+ 2499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:16.179929 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:16.179941 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:16.180384 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:16.180684 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:16.180688 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:16.180692 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:16.180890 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:16.181490 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:16.181500 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:16.181862 (+ 362us) client_negotiation.cc:769] Sending connection context
0927 10:46:16.182077 (+ 215us) client_negotiation.cc:240] Negotiation successful
0927 10:46:16.182622 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":57,"thread_start_us":158,"threads_started":1}
I20260927 10:46:16.184412 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:16.169088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58770 (local address 127.0.98.60:44805)
0927 10:46:16.169411 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:16.169421 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:16.169623 (+ 202us) server_negotiation.cc:407] Connection header received
0927 10:46:16.170331 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:16.170358 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:16.170646 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:16.170838 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:16.171712 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.173020 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.174161 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.174701 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.174887 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:16.178272 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:16.178282 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:16.178299 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:16.178329 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:16.181591 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:16.182239 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:16.182249 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:16.182253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:16.182338 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:16.182851 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:16.182854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:16.182856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:16.183162 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:16.183348 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:16.184054 (+ 706us) server_negotiation.cc:299] Negotiation successful
0927 10:46:16.184222 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":235,"thread_start_us":118,"threads_started":1}
I20260927 10:46:16.186837 13925 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35013 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:46:16 dist-test-slave-dzhw krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 10:46:16 dist-test-slave-dzhw krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 10:46:16 dist-test-slave-dzhw krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 10:46:16.195130 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:16.183930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44805 (local address 127.0.0.1:58780)
0927 10:46:16.184312 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:16.184326 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:16.184397 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:16.185098 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:16.185104 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:16.185319 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:16.185427 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:16.185436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.187703 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.187707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:16.188478 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:16.188486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.188981 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.188984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:16.189067 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:16.189527 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:16.189538 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:16.190843 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:16.193362 (+ 2519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:16.193365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:16.193368 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:16.193641 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:16.193915 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:16.193918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:16.193920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:16.194004 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:16.194464 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:16.194469 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:16.194596 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 10:46:16.194732 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 10:46:16.194922 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303}
I20260927 10:46:16.195441 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:16.184542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58780 (local address 127.0.98.60:44805)
0927 10:46:16.184869 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:16.184877 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:16.184888 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:46:16.184920 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:16.184922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:16.184962 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:16.185048 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:16.185564 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.187284 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.188652 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.188871 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.188980 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:16.190972 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:16.190975 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:16.190978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:16.191000 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:16.193255 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:16.193748 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:16.193752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:16.193753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:16.193837 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:16.194119 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:16.194123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:16.194125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:16.194329 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:16.194404 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:16.195185 (+ 781us) server_negotiation.cc:299] Negotiation successful
0927 10:46:16.195314 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":148,"threads_started":1}
I20260927 10:46:16.196170 13849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } attempt: 3
I20260927 10:46:16.197629 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:16.187483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50688 (local address 127.0.98.61:35013)
0927 10:46:16.187600 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:16.187604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:16.187620 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:46:16.187665 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:16.187668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:16.187726 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:16.187806 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:16.188232 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.189000 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.190363 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.190762 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.190866 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:16.193906 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:16.193909 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:16.193913 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:16.193947 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:16.195762 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:16.196285 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:16.196288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:16.196290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:16.196371 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:16.196682 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:16.196686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:16.196688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:16.196923 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:16.197022 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:16.197351 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 10:46:16.197462 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":46}
I20260927 10:46:16.198086 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:16.187257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35013 (local address 127.0.0.1:50688)
0927 10:46:16.187499 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:16.187512 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:16.187580 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:16.187829 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:16.187832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:16.188005 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:16.188104 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:16.188110 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.189130 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.189133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:16.190245 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:16.190255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.190882 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.190888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:16.190997 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:16.191486 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:16.191504 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:16.193764 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:16.195864 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:16.195866 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:16.195868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:16.196164 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:16.196493 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:16.196496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:16.196497 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:16.196582 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:16.197059 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:16.197062 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:16.197174 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 10:46:16.197817 (+ 643us) client_negotiation.cc:240] Negotiation successful
0927 10:46:16.197947 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260927 10:46:16.198522 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:16.187445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47554 (local address 127.0.98.62:42999)
0927 10:46:16.187564 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:16.187568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:16.187731 (+ 163us) server_negotiation.cc:407] Connection header received
0927 10:46:16.187893 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:16.187896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:16.187932 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:16.187980 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:16.188938 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.189657 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.190739 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.191044 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.191167 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:16.193997 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:16.194000 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:16.194003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:16.194023 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:16.195831 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:16.197109 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:16.197112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:16.197114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:16.197197 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:16.197508 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:16.197511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:16.197513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:16.197712 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:16.197891 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:16.198248 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 10:46:16.198365 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:46:16.199717 13774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89c6a65a734c4fb98df3270a485e8643: Bootstrap starting.
Sep 27 10:46:16 dist-test-slave-dzhw krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 10:46:16.202527 13774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89c6a65a734c4fb98df3270a485e8643: Neither blocks nor log segments found. Creating new log.
I20260927 10:46:16.203466 13774 log.cc:824] T 00000000000000000000000000000000 P 89c6a65a734c4fb98df3270a485e8643: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:16.204629 13925 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:46:16.205843 13774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89c6a65a734c4fb98df3270a485e8643: No bootstrap required, opened a new log
I20260927 10:46:16.206055 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:16.196576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44805 (local address 127.0.0.1:58784)
0927 10:46:16.196764 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:16.196780 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:16.196866 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:16.197173 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:16.197176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:16.197352 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:16.197449 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:16.197455 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.198891 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.198895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:16.199670 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:16.199680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.200235 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.200238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:16.200344 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:16.200825 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:16.200841 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:16.202371 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:16.204420 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:16.204427 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:16.204432 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:16.204677 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:16.205015 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:16.205018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:16.205019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:16.205112 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:16.205570 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:16.205573 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:16.205668 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 10:46:16.205750 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 10:46:16.205882 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
Sep 27 10:46:16 dist-test-slave-dzhw krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 10:46:16.206957 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:16.196803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58784 (local address 127.0.98.60:44805)
0927 10:46:16.196959 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:16.196965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:16.196978 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:46:16.197013 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:16.197016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:16.197058 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:16.197549 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:16.197643 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.198780 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.199790 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.200093 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.200232 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:16.202583 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:16.202587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:16.202589 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:16.202617 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:16.204267 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:16.204834 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:16.204837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:16.204838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:16.204917 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:16.205229 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:16.205232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:16.205233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:16.205429 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:16.206631 (+ 1202us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:16.206707 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 10:46:16.206825 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 10:46:16.210197 13849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e: Bootstrap starting.
I20260927 10:46:16.210106 13774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89c6a65a734c4fb98df3270a485e8643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6a65a734c4fb98df3270a485e8643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "26cc089c608342e28407fd9bf859a85e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35013 } } peers { permanent_uuid: "8f802173746b4ecbafb9b6b54e6d4777" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } }
I20260927 10:46:16.210340 13774 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89c6a65a734c4fb98df3270a485e8643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:46:16.210383 13774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89c6a65a734c4fb98df3270a485e8643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c6a65a734c4fb98df3270a485e8643, State: Initialized, Role: FOLLOWER
Sep 27 10:46:16 dist-test-slave-dzhw krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 10:46:16.210752 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:16.200055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50690 (local address 127.0.98.61:35013)
0927 10:46:16.200163 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:16.200167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:16.200235 (+ 68us) server_negotiation.cc:407] Connection header received
0927 10:46:16.200394 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:16.200397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:16.200439 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:16.200490 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:16.201356 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.201874 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.202782 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.203170 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.203285 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:16.206928 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:16.206934 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:16.206941 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:16.206972 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:16.208770 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:16.209234 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:16.209237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:16.209239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:16.209323 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:16.209682 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:16.209687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:16.209692 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:16.209897 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:16.210056 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:16.210465 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 10:46:16.210591 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 10:46:16.210932 13774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89c6a65a734c4fb98df3270a485e8643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6a65a734c4fb98df3270a485e8643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "26cc089c608342e28407fd9bf859a85e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35013 } } peers { permanent_uuid: "8f802173746b4ecbafb9b6b54e6d4777" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } }
I20260927 10:46:16.212288 13774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89c6a65a734c4fb98df3270a485e8643 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:46:16.212940 13849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e: Neither blocks nor log segments found. Creating new log.
W20260927 10:46:16.213721 13727 tablet.cc:2369] T 00000000000000000000000000000000 P 89c6a65a734c4fb98df3270a485e8643: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:46:16.213850 13849 log.cc:824] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:16.214754 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:16.205155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58800 (local address 127.0.98.60:44805)
0927 10:46:16.205253 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:16.205258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:16.205362 (+ 104us) server_negotiation.cc:407] Connection header received
0927 10:46:16.205468 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:16.205472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:16.205505 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:16.205569 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:16.206087 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.206900 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.208524 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.208902 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.208991 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:16.211218 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:16.211223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:16.211225 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:16.211249 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:16.212876 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:16.213367 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:16.213369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:16.213371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:16.213443 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:16.213707 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:16.213710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:16.213714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:16.213905 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:16.213991 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:16.214514 (+ 523us) server_negotiation.cc:299] Negotiation successful
0927 10:46:16.214620 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 10:46:16.215041 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:46:16.215056 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:16.205124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44805 (local address 127.0.0.1:58800)
0927 10:46:16.205295 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:16.205307 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:16.205400 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:16.205682 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:16.205687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:16.205864 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:16.205944 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:16.205951 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.206997 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.207001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:16.208398 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:16.208410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.209092 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.209096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:16.209199 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:16.209643 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:16.209662 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:16.211070 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:16.213015 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:16.213018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:16.213023 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:16.213269 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:16.213533 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:16.213536 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:16.213539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:16.213619 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:16.214071 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:16.214074 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:16.214169 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 10:46:16.214270 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 10:46:16.214396 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260927 10:46:16.217176 13849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e: No bootstrap required, opened a new log
I20260927 10:46:16.217850 13925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f802173746b4ecbafb9b6b54e6d4777: Bootstrap starting.
W20260927 10:46:16.218915 13944 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 89c6a65a734c4fb98df3270a485e8643: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:46:16.218967 13944 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:46:16.220494 13925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f802173746b4ecbafb9b6b54e6d4777: Neither blocks nor log segments found. Creating new log.
I20260927 10:46:16.221221 13849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6a65a734c4fb98df3270a485e8643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "26cc089c608342e28407fd9bf859a85e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35013 } } peers { permanent_uuid: "8f802173746b4ecbafb9b6b54e6d4777" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } }
I20260927 10:46:16.221359 13925 log.cc:824] T 00000000000000000000000000000000 P 8f802173746b4ecbafb9b6b54e6d4777: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:16.221432 13849 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:46:16.221477 13849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26cc089c608342e28407fd9bf859a85e, State: Initialized, Role: FOLLOWER
I20260927 10:46:16.218822 13706 master_runner.cc:186] Error getting master registration for 127.0.98.61:35013: OK, instance_id { permanent_uuid: "26cc089c608342e28407fd9bf859a85e" instance_seqno: 1790505975941985 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 10:46:16.221582 13706 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:46:16.221992 13849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6a65a734c4fb98df3270a485e8643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "26cc089c608342e28407fd9bf859a85e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35013 } } peers { permanent_uuid: "8f802173746b4ecbafb9b6b54e6d4777" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } }
I20260927 10:46:16.222540 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6a65a734c4fb98df3270a485e8643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "26cc089c608342e28407fd9bf859a85e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35013 } } peers { permanent_uuid: "8f802173746b4ecbafb9b6b54e6d4777" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } } }
I20260927 10:46:16.222743 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:46:16.223439 13849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:46:16.223842 13925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f802173746b4ecbafb9b6b54e6d4777: No bootstrap required, opened a new log
I20260927 10:46:16.226586 13932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89c6a65a734c4fb98df3270a485e8643 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6a65a734c4fb98df3270a485e8643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "26cc089c608342e28407fd9bf859a85e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35013 } } peers { permanent_uuid: "8f802173746b4ecbafb9b6b54e6d4777" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } } }
I20260927 10:46:16.226748 13932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89c6a65a734c4fb98df3270a485e8643 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:46:16.227916 13925 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f802173746b4ecbafb9b6b54e6d4777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6a65a734c4fb98df3270a485e8643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "26cc089c608342e28407fd9bf859a85e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35013 } } peers { permanent_uuid: "8f802173746b4ecbafb9b6b54e6d4777" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } }
I20260927 10:46:16.228533 13925 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f802173746b4ecbafb9b6b54e6d4777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:46:16.228600 13925 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f802173746b4ecbafb9b6b54e6d4777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f802173746b4ecbafb9b6b54e6d4777, State: Initialized, Role: FOLLOWER
I20260927 10:46:16.229126 13925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f802173746b4ecbafb9b6b54e6d4777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6a65a734c4fb98df3270a485e8643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "26cc089c608342e28407fd9bf859a85e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35013 } } peers { permanent_uuid: "8f802173746b4ecbafb9b6b54e6d4777" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } }
I20260927 10:46:16.231087 13925 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f802173746b4ecbafb9b6b54e6d4777 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:46:16.233711 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:16.222083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47566 (local address 127.0.98.62:42999)
0927 10:46:16.222233 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:16.222237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:16.223627 (+ 1390us) server_negotiation.cc:407] Connection header received
0927 10:46:16.223795 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:16.223800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:16.223856 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:16.223935 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:16.225075 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.225848 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.227086 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.227396 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.227474 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:16.229606 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:16.229611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:16.229613 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:16.229640 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:16.231550 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:16.232082 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:16.232085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:16.232087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:16.232171 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:16.232467 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:16.232471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:16.232474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:16.232692 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:16.232802 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:16.233169 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 10:46:16.233297 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 10:46:16.231142 13781 master_runner.cc:186] Error getting master registration for 127.0.98.60:44805: OK, instance_id { permanent_uuid: "8f802173746b4ecbafb9b6b54e6d4777" instance_seqno: 1790505976156385 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 10:46:16.235507 13781 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 10:46:16.236886 13977 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f802173746b4ecbafb9b6b54e6d4777: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:46:16.236964 13977 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:46:16.237280 13966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f802173746b4ecbafb9b6b54e6d4777 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6a65a734c4fb98df3270a485e8643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "26cc089c608342e28407fd9bf859a85e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35013 } } peers { permanent_uuid: "8f802173746b4ecbafb9b6b54e6d4777" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } } }
I20260927 10:46:16.237406 13966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f802173746b4ecbafb9b6b54e6d4777 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:46:16.240221 13856 master_runner.cc:263] No leader master found from master 8f802173746b4ecbafb9b6b54e6d4777
I20260927 10:46:16.240289 13856 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 10:46:16.244177 13965 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:46:16.244254 13965 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:46:16.244608 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:16.223292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50700 (local address 127.0.98.61:35013)
0927 10:46:16.223390 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:16.223394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:16.234526 (+ 11132us) server_negotiation.cc:407] Connection header received
0927 10:46:16.234702 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:16.234706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:16.234754 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:16.234802 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:16.235796 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.236604 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.237848 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.238222 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.238339 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:16.240248 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:16.240253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:16.240255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:16.240546 (+ 291us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:16.242770 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:16.243265 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:16.243269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:16.243270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:16.243353 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:16.243648 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:16.243651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:16.243653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:16.243867 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:16.243968 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:16.244333 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 10:46:16.244450 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 10:46:16.245337 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:16.222882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58808 (local address 127.0.98.60:44805)
0927 10:46:16.222995 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:16.222999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:16.230903 (+ 7904us) server_negotiation.cc:407] Connection header received
0927 10:46:16.235766 (+ 4863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:16.235772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:16.235821 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:16.235871 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:16.236738 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.237480 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.239078 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:16.239379 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:16.239455 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:16.241514 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:16.241518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:16.241520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:16.241545 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:16.243189 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:16.243658 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:16.243663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:16.243665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:16.243750 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:16.244055 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:16.244059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:16.244060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:16.244271 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:16.244371 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:16.245071 (+ 700us) server_negotiation.cc:299] Negotiation successful
0927 10:46:16.245188 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 10:46:16.282738 13951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:46:16.282948 13951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6a65a734c4fb98df3270a485e8643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "26cc089c608342e28407fd9bf859a85e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35013 } } peers { permanent_uuid: "8f802173746b4ecbafb9b6b54e6d4777" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } }
I20260927 10:46:16.284574 13748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26cc089c608342e28407fd9bf859a85e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89c6a65a734c4fb98df3270a485e8643" is_pre_election: true
I20260927 10:46:16.284904 13748 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 89c6a65a734c4fb98df3270a485e8643 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26cc089c608342e28407fd9bf859a85e in term 0.
I20260927 10:46:16.285356 13799 leader_election.cc:302] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26cc089c608342e28407fd9bf859a85e, 89c6a65a734c4fb98df3270a485e8643; no voters:
I20260927 10:46:16.286425 13899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26cc089c608342e28407fd9bf859a85e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f802173746b4ecbafb9b6b54e6d4777" is_pre_election: true
I20260927 10:46:16.286770 13899 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8f802173746b4ecbafb9b6b54e6d4777 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26cc089c608342e28407fd9bf859a85e in term 0.
I20260927 10:46:16.289902 13951 leader_election.cc:288] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89c6a65a734c4fb98df3270a485e8643 (127.0.98.62:42999), 8f802173746b4ecbafb9b6b54e6d4777 (127.0.98.60:44805)
I20260927 10:46:16.290238 13951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:46:16.290333 13951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:46:16.290378 13951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:46:16.291760 13951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6a65a734c4fb98df3270a485e8643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "26cc089c608342e28407fd9bf859a85e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35013 } } peers { permanent_uuid: "8f802173746b4ecbafb9b6b54e6d4777" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } }
I20260927 10:46:16.292461 13748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26cc089c608342e28407fd9bf859a85e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89c6a65a734c4fb98df3270a485e8643"
I20260927 10:46:16.292606 13748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89c6a65a734c4fb98df3270a485e8643 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:46:16.293954 13748 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 89c6a65a734c4fb98df3270a485e8643 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26cc089c608342e28407fd9bf859a85e in term 1.
I20260927 10:46:16.294314 13799 leader_election.cc:302] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26cc089c608342e28407fd9bf859a85e, 89c6a65a734c4fb98df3270a485e8643; no voters:
I20260927 10:46:16.294910 13899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26cc089c608342e28407fd9bf859a85e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f802173746b4ecbafb9b6b54e6d4777"
I20260927 10:46:16.295065 13899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f802173746b4ecbafb9b6b54e6d4777 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:46:16.296847 13899 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8f802173746b4ecbafb9b6b54e6d4777 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26cc089c608342e28407fd9bf859a85e in term 1.
I20260927 10:46:16.299710 13951 leader_election.cc:288] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [CANDIDATE]: Term 1 election: Requested vote from peers 89c6a65a734c4fb98df3270a485e8643 (127.0.98.62:42999), 8f802173746b4ecbafb9b6b54e6d4777 (127.0.98.60:44805)
I20260927 10:46:16.299894 13951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:46:16.299983 13951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [term 1 LEADER]: Becoming Leader. State: Replica: 26cc089c608342e28407fd9bf859a85e, State: Running, Role: LEADER
I20260927 10:46:16.300364 13951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6a65a734c4fb98df3270a485e8643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "26cc089c608342e28407fd9bf859a85e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35013 } } peers { permanent_uuid: "8f802173746b4ecbafb9b6b54e6d4777" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } }
I20260927 10:46:16.302240 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26cc089c608342e28407fd9bf859a85e. Latest consensus state: current_term: 1 leader_uuid: "26cc089c608342e28407fd9bf859a85e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6a65a734c4fb98df3270a485e8643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "26cc089c608342e28407fd9bf859a85e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35013 } } peers { permanent_uuid: "8f802173746b4ecbafb9b6b54e6d4777" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } } }
I20260927 10:46:16.302354 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [sys.catalog]: This master's current role is: LEADER
I20260927 10:46:16.302753 13982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:46:16.303705 13982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:46:16.311059 13899 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8f802173746b4ecbafb9b6b54e6d4777 [term 1 FOLLOWER]: Refusing update from remote peer 26cc089c608342e28407fd9bf859a85e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:46:16.312314 13980 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f802173746b4ecbafb9b6b54e6d4777" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:46:16.319888 13966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f802173746b4ecbafb9b6b54e6d4777 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26cc089c608342e28407fd9bf859a85e. Latest consensus state: current_term: 1 leader_uuid: "26cc089c608342e28407fd9bf859a85e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6a65a734c4fb98df3270a485e8643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "26cc089c608342e28407fd9bf859a85e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35013 } } peers { permanent_uuid: "8f802173746b4ecbafb9b6b54e6d4777" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } } }
I20260927 10:46:16.320063 13966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f802173746b4ecbafb9b6b54e6d4777 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:46:16.321249 13748 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 89c6a65a734c4fb98df3270a485e8643 [term 1 FOLLOWER]: Refusing update from remote peer 26cc089c608342e28407fd9bf859a85e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:46:16.321910 13951 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [LEADER]: Connected to new peer: Peer: permanent_uuid: "89c6a65a734c4fb98df3270a485e8643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:46:16.326578 13932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89c6a65a734c4fb98df3270a485e8643 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26cc089c608342e28407fd9bf859a85e. Latest consensus state: current_term: 1 leader_uuid: "26cc089c608342e28407fd9bf859a85e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6a65a734c4fb98df3270a485e8643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "26cc089c608342e28407fd9bf859a85e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35013 } } peers { permanent_uuid: "8f802173746b4ecbafb9b6b54e6d4777" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } } }
I20260927 10:46:16.326736 13932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89c6a65a734c4fb98df3270a485e8643 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:46:16.336166 13980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26cc089c608342e28407fd9bf859a85e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6a65a734c4fb98df3270a485e8643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "26cc089c608342e28407fd9bf859a85e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35013 } } peers { permanent_uuid: "8f802173746b4ecbafb9b6b54e6d4777" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } } }
I20260927 10:46:16.336355 13980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [sys.catalog]: This master's current role is: LEADER
I20260927 10:46:16.337718 13932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89c6a65a734c4fb98df3270a485e8643 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26cc089c608342e28407fd9bf859a85e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6a65a734c4fb98df3270a485e8643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "26cc089c608342e28407fd9bf859a85e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35013 } } peers { permanent_uuid: "8f802173746b4ecbafb9b6b54e6d4777" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } } }
I20260927 10:46:16.337864 13932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89c6a65a734c4fb98df3270a485e8643 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:46:16.338532 13983 mvcc.cc:205] Tried to move back new op lower bound from 7333912478961741824 to 7333912478929489920. Current Snapshot: MvccSnapshot[applied={T|T < 7333912478961741824}]
I20260927 10:46:16.340442 13986 mvcc.cc:205] Tried to move back new op lower bound from 7333912478961741824 to 7333912478929489920. Current Snapshot: MvccSnapshot[applied={T|T < 7333912478961741824}]
I20260927 10:46:16.341374 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26cc089c608342e28407fd9bf859a85e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6a65a734c4fb98df3270a485e8643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "26cc089c608342e28407fd9bf859a85e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35013 } } peers { permanent_uuid: "8f802173746b4ecbafb9b6b54e6d4777" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } } }
I20260927 10:46:16.341527 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e [sys.catalog]: This master's current role is: LEADER
I20260927 10:46:16.344900 13966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f802173746b4ecbafb9b6b54e6d4777 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26cc089c608342e28407fd9bf859a85e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89c6a65a734c4fb98df3270a485e8643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "26cc089c608342e28407fd9bf859a85e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35013 } } peers { permanent_uuid: "8f802173746b4ecbafb9b6b54e6d4777" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44805 } } }
I20260927 10:46:16.345041 13966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f802173746b4ecbafb9b6b54e6d4777 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:46:16.346073 13984 mvcc.cc:205] Tried to move back new op lower bound from 7333912478961741824 to 7333912478929489920. Current Snapshot: MvccSnapshot[applied={T|T < 7333912478961741824}]
I20260927 10:46:16.351732 13982 catalog_manager.cc:1347] Generated new cluster ID: fcc76b426ccc4ac6af89758f219d7dcf
I20260927 10:46:16.351824 13982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:46:16.386576 13982 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:46:16.387941 13982 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:46:16.407678 13982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26cc089c608342e28407fd9bf859a85e: Generated new TSK 0
I20260927 10:46:16.408594 13982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:46:17.220227 13944 catalog_manager.cc:1259] Loaded cluster ID: fcc76b426ccc4ac6af89758f219d7dcf
I20260927 10:46:17.220315 13944 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 89c6a65a734c4fb98df3270a485e8643: loading cluster ID for follower catalog manager: success
I20260927 10:46:17.222111 13944 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 89c6a65a734c4fb98df3270a485e8643: acquiring CA information for follower catalog manager: success
I20260927 10:46:17.223392 13944 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 89c6a65a734c4fb98df3270a485e8643: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:46:17.238525 13977 catalog_manager.cc:1259] Loaded cluster ID: fcc76b426ccc4ac6af89758f219d7dcf
I20260927 10:46:17.238602 13977 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8f802173746b4ecbafb9b6b54e6d4777: loading cluster ID for follower catalog manager: success
I20260927 10:46:17.240265 13977 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8f802173746b4ecbafb9b6b54e6d4777: acquiring CA information for follower catalog manager: success
I20260927 10:46:17.241446 13977 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8f802173746b4ecbafb9b6b54e6d4777: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:46:17.473024 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 13706
I20260927 10:46:17.479359 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 13781
I20260927 10:46:17.485176 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 13856
2026-09-27T10:46:17Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3100 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:46:17 dist-test-slave-dzhw krb5kdc[13997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:46:17 dist-test-slave-dzhw krb5kdc[13997](info): set up 2 sockets
Sep 27 10:46:17 dist-test-slave-dzhw krb5kdc[13997](info): commencing operation
krb5kdc: starting...
W20260927 10:46:18.530695 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:46:18 dist-test-slave-dzhw krb5kdc[13997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:46:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:46:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:46:18.651337 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36815
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:46:18.762955 14010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:18.763243 14010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:18.763286 14010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:18.767565 14010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:46:18.767627 14010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:18.767647 14010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:18.767663 14010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:46:18.767702 14010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:46:18.773098 14010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:18.774215 14010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:18.775275 14010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:18.783869 14018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:18.783907 14010 server_base.cc:1034] running on GCE node
W20260927 10:46:18.783880 14015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:18.783866 14016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:18.784610 14010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:18.785823 14010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:18.786994 14010 hybrid_clock.cc:648] HybridClock initialized: now 1790505978786986 us; error 39 us; skew 500 ppm
Sep 27 10:46:18 dist-test-slave-dzhw krb5kdc[13997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:18.789649 14010 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:46:18.790527 14010 webserver.cc:460] Webserver started at http://127.0.98.62:43781/ using document root <none> and password file <none>
I20260927 10:46:18.791039 14010 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:18.791105 14010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:18.791268 14010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:46:18.793316 14010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "8d2fa8da3d5c4172ab3634f083c5fd5f"
format_stamp: "Formatted at 2026-09-27 10:46:18 on dist-test-slave-dzhw"
server_key: "e080889172a2a56946981feadf699595"
server_key_iv: "bbefde9132c9b408d10c84bd17697df3"
server_key_version: "encryptionkey@0"
I20260927 10:46:18.793843 14010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "8d2fa8da3d5c4172ab3634f083c5fd5f"
format_stamp: "Formatted at 2026-09-27 10:46:18 on dist-test-slave-dzhw"
server_key: "e080889172a2a56946981feadf699595"
server_key_iv: "bbefde9132c9b408d10c84bd17697df3"
server_key_version: "encryptionkey@0"
I20260927 10:46:18.797559 14010 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 10:46:18.800062 14025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:18.801388 14010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:46:18.801553 14010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "8d2fa8da3d5c4172ab3634f083c5fd5f"
format_stamp: "Formatted at 2026-09-27 10:46:18 on dist-test-slave-dzhw"
server_key: "e080889172a2a56946981feadf699595"
server_key_iv: "bbefde9132c9b408d10c84bd17697df3"
server_key_version: "encryptionkey@0"
I20260927 10:46:18.801671 14010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:18.820977 14010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:18.823904 14010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:18.824110 14010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:18.830854 14010 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36815
I20260927 10:46:18.830911 14077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36815 every 8 connection(s)
I20260927 10:46:18.831840 14010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:46:18.834664 14078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:46:18.838249 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 14010
I20260927 10:46:18.838375 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:46:18.838686 392 external_mini_cluster.cc:1428] Setting key caaaa2bb58888f436cb235c0f543bfbf
I20260927 10:46:18.840740 14078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d2fa8da3d5c4172ab3634f083c5fd5f: Bootstrap starting.
I20260927 10:46:18.842924 14078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d2fa8da3d5c4172ab3634f083c5fd5f: Neither blocks nor log segments found. Creating new log.
I20260927 10:46:18.843631 14078 log.cc:824] T 00000000000000000000000000000000 P 8d2fa8da3d5c4172ab3634f083c5fd5f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:18.845388 14078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d2fa8da3d5c4172ab3634f083c5fd5f: No bootstrap required, opened a new log
Sep 27 10:46:18 dist-test-slave-dzhw krb5kdc[13997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:46:18.849236 14078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d2fa8da3d5c4172ab3634f083c5fd5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2fa8da3d5c4172ab3634f083c5fd5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36815 } }
I20260927 10:46:18.849481 14078 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d2fa8da3d5c4172ab3634f083c5fd5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:46:18.849556 14078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d2fa8da3d5c4172ab3634f083c5fd5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d2fa8da3d5c4172ab3634f083c5fd5f, State: Initialized, Role: FOLLOWER
I20260927 10:46:18.850013 14078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d2fa8da3d5c4172ab3634f083c5fd5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2fa8da3d5c4172ab3634f083c5fd5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36815 } }
I20260927 10:46:18.850157 14078 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d2fa8da3d5c4172ab3634f083c5fd5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:46:18.850224 14078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d2fa8da3d5c4172ab3634f083c5fd5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:46:18.850327 14078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d2fa8da3d5c4172ab3634f083c5fd5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:46:18.851204 14078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d2fa8da3d5c4172ab3634f083c5fd5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2fa8da3d5c4172ab3634f083c5fd5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36815 } }
I20260927 10:46:18.851543 14078 leader_election.cc:302] T 00000000000000000000000000000000 P 8d2fa8da3d5c4172ab3634f083c5fd5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d2fa8da3d5c4172ab3634f083c5fd5f; no voters:
I20260927 10:46:18.851881 14078 leader_election.cc:288] T 00000000000000000000000000000000 P 8d2fa8da3d5c4172ab3634f083c5fd5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:46:18.851953 14083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d2fa8da3d5c4172ab3634f083c5fd5f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:46:18.852149 14083 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d2fa8da3d5c4172ab3634f083c5fd5f [term 1 LEADER]: Becoming Leader. State: Replica: 8d2fa8da3d5c4172ab3634f083c5fd5f, State: Running, Role: LEADER
I20260927 10:46:18.852489 14083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d2fa8da3d5c4172ab3634f083c5fd5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2fa8da3d5c4172ab3634f083c5fd5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36815 } }
I20260927 10:46:18.852895 14081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:18.840079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54550 (local address 127.0.98.62:36815)
0927 10:46:18.840515 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:18.840530 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:18.840564 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:46:18.841250 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:18.841267 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:18.841605 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:18.841930 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:18.842755 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:18.843819 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:18.844654 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:18.845116 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:18.845278 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:18.847814 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:18.847819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:18.847831 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:18.847857 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:18.849754 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:18.850192 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:18.850198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:18.850206 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:18.850387 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:18.850685 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:18.850689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:18.850691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:18.851084 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:18.851307 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:18.851558 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 10:46:18.851877 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":131,"threads_started":1}
I20260927 10:46:18.853152 14078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d2fa8da3d5c4172ab3634f083c5fd5f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:46:18.855851 14084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d2fa8da3d5c4172ab3634f083c5fd5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d2fa8da3d5c4172ab3634f083c5fd5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2fa8da3d5c4172ab3634f083c5fd5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36815 } } }
I20260927 10:46:18.855970 14084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d2fa8da3d5c4172ab3634f083c5fd5f [sys.catalog]: This master's current role is: LEADER
I20260927 10:46:18.855976 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:46:18.856828 14085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d2fa8da3d5c4172ab3634f083c5fd5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d2fa8da3d5c4172ab3634f083c5fd5f. Latest consensus state: current_term: 1 leader_uuid: "8d2fa8da3d5c4172ab3634f083c5fd5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2fa8da3d5c4172ab3634f083c5fd5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36815 } } }
I20260927 10:46:18.856926 14085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d2fa8da3d5c4172ab3634f083c5fd5f [sys.catalog]: This master's current role is: LEADER
W20260927 10:46:18.858059 14097 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8d2fa8da3d5c4172ab3634f083c5fd5f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:46:18.858140 14097 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:46:18.858316 14098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:46:18.858997 14098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:46:18.864167 14098 catalog_manager.cc:1347] Generated new cluster ID: 960482e616cd44ee98782757c705c792
I20260927 10:46:18.864236 14098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:46:18.869704 14098 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:46:18.870746 14098 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:46:18.876502 14098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d2fa8da3d5c4172ab3634f083c5fd5f: Generated new TSK 0
I20260927 10:46:18.877195 14098 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:46:18.911868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 14010
2026-09-27T10:46:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1424 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:46:18 dist-test-slave-dzhw krb5kdc[14105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:46:18 dist-test-slave-dzhw krb5kdc[14105](info): set up 2 sockets
Sep 27 10:46:18 dist-test-slave-dzhw krb5kdc[14105](info): commencing operation
krb5kdc: starting...
W20260927 10:46:19.958681 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:46:20 dist-test-slave-dzhw krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:46:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:46:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:46:20.082849 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41583
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:46:20.198529 14118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:20.198815 14118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:20.198858 14118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:20.202988 14118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:46:20.203066 14118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:20.203087 14118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:20.203166 14118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:46:20.203200 14118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:46:20.208863 14118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:20.209990 14118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:20.211057 14118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:20.219444 14123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:20.219447 14126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:20.219475 14118 server_base.cc:1034] running on GCE node
W20260927 10:46:20.219666 14124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:20.220311 14118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:20.221397 14118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:20.222554 14118 hybrid_clock.cc:648] HybridClock initialized: now 1790505980222541 us; error 27 us; skew 500 ppm
Sep 27 10:46:20 dist-test-slave-dzhw krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:20.230109 14118 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:46:20.231154 14118 webserver.cc:460] Webserver started at http://127.0.98.62:38861/ using document root <none> and password file <none>
I20260927 10:46:20.231756 14118 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:20.231851 14118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:20.232079 14118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:46:20.234197 14118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "74aa3fb48ada44db96baf018b173eefb"
format_stamp: "Formatted at 2026-09-27 10:46:20 on dist-test-slave-dzhw"
server_key: "8cf99d10a6d612f121b97bfea5bd45ed"
server_key_iv: "ed557d852b6de1e1fed9a82bd262e691"
server_key_version: "encryptionkey@0"
I20260927 10:46:20.234838 14118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "74aa3fb48ada44db96baf018b173eefb"
format_stamp: "Formatted at 2026-09-27 10:46:20 on dist-test-slave-dzhw"
server_key: "8cf99d10a6d612f121b97bfea5bd45ed"
server_key_iv: "ed557d852b6de1e1fed9a82bd262e691"
server_key_version: "encryptionkey@0"
I20260927 10:46:20.238735 14118 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:46:20.241315 14133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:20.242673 14118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:46:20.242822 14118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "74aa3fb48ada44db96baf018b173eefb"
format_stamp: "Formatted at 2026-09-27 10:46:20 on dist-test-slave-dzhw"
server_key: "8cf99d10a6d612f121b97bfea5bd45ed"
server_key_iv: "ed557d852b6de1e1fed9a82bd262e691"
server_key_version: "encryptionkey@0"
I20260927 10:46:20.242947 14118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:20.255319 14118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:20.258752 14118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:20.258971 14118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:20.266672 14118 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41583
I20260927 10:46:20.266989 14185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41583 every 8 connection(s)
I20260927 10:46:20.267765 14118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:46:20.269129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 14118
I20260927 10:46:20.269289 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:46:20.269594 392 external_mini_cluster.cc:1428] Setting key a6d3b73a8cfc38db0b9351d48f976fc7
I20260927 10:46:20.270979 14186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:46:20.278021 14186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb: Bootstrap starting.
Sep 27 10:46:20 dist-test-slave-dzhw krb5kdc[14105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:46:20.280939 14186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb: Neither blocks nor log segments found. Creating new log.
I20260927 10:46:20.281980 14186 log.cc:824] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:20.284435 14186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb: No bootstrap required, opened a new log
I20260927 10:46:20.285876 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:20.271178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48298 (local address 127.0.98.62:41583)
0927 10:46:20.271611 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:20.271630 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:20.271672 (+ 42us) server_negotiation.cc:407] Connection header received
0927 10:46:20.272561 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:20.272585 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:20.273013 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:20.273409 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:20.274096 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:20.275622 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:20.276500 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:20.277231 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:20.277381 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:20.280248 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:20.280254 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:20.280272 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:20.280308 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:20.282688 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:20.283185 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:20.283192 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:20.283198 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:20.283332 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:20.283620 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:20.283622 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:20.283624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:20.284118 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:20.284306 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:20.284628 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 10:46:20.284941 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":121,"threads_started":1}
I20260927 10:46:20.288810 14186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74aa3fb48ada44db96baf018b173eefb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41583 } }
I20260927 10:46:20.289083 14186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:46:20.289146 14186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74aa3fb48ada44db96baf018b173eefb, State: Initialized, Role: FOLLOWER
I20260927 10:46:20.289585 14186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74aa3fb48ada44db96baf018b173eefb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41583 } }
I20260927 10:46:20.289732 14186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:46:20.289813 14186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:46:20.289911 14186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:46:20.290920 14186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74aa3fb48ada44db96baf018b173eefb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41583 } }
I20260927 10:46:20.291260 14186 leader_election.cc:302] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74aa3fb48ada44db96baf018b173eefb; no voters:
I20260927 10:46:20.291570 14186 leader_election.cc:288] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:46:20.291783 14191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:46:20.292006 14191 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [term 1 LEADER]: Becoming Leader. State: Replica: 74aa3fb48ada44db96baf018b173eefb, State: Running, Role: LEADER
I20260927 10:46:20.292313 14191 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74aa3fb48ada44db96baf018b173eefb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41583 } }
I20260927 10:46:20.292867 14186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:46:20.293919 14193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74aa3fb48ada44db96baf018b173eefb. Latest consensus state: current_term: 1 leader_uuid: "74aa3fb48ada44db96baf018b173eefb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74aa3fb48ada44db96baf018b173eefb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41583 } } }
I20260927 10:46:20.294036 14193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [sys.catalog]: This master's current role is: LEADER
I20260927 10:46:20.294384 14192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74aa3fb48ada44db96baf018b173eefb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74aa3fb48ada44db96baf018b173eefb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41583 } } }
I20260927 10:46:20.294589 14192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [sys.catalog]: This master's current role is: LEADER
I20260927 10:46:20.295291 14200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:46:20.297852 14200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:46:20.303066 14200 catalog_manager.cc:1347] Generated new cluster ID: 36a62e5caf1643839c0213c6011b43b2
I20260927 10:46:20.303141 14200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:46:20.323486 14200 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:46:20.324826 14200 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:46:20.330264 14200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb: Generated new TSK 0
I20260927 10:46:20.331176 14200 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 10:46:20.362485 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41583
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:46:20.476092 14212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:20.476434 14212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:20.476522 14212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:20.481062 14212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:20.481150 14212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:20.481252 14212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:46:20.487147 14212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:20.488368 14212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:20.489459 14212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:20.499540 14217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:20.499652 14212 server_base.cc:1034] running on GCE node
W20260927 10:46:20.499534 14218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:20.499534 14220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:20.500288 14212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:20.500982 14212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:20.502141 14212 hybrid_clock.cc:648] HybridClock initialized: now 1790505980502121 us; error 33 us; skew 500 ppm
Sep 27 10:46:20 dist-test-slave-dzhw krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:20.504899 14212 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 10:46:20.505810 14212 webserver.cc:460] Webserver started at http://127.0.98.1:35261/ using document root <none> and password file <none>
I20260927 10:46:20.506395 14212 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:20.506548 14212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:20.506778 14212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:46:20.508929 14212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "b80217ddf067482793d1d3ab27c676d7"
format_stamp: "Formatted at 2026-09-27 10:46:20 on dist-test-slave-dzhw"
server_key: "bde0fd74426ba689bfe3c8fe68596407"
server_key_iv: "e1ca6d8ea164cc9a624f710a00c48a58"
server_key_version: "encryptionkey@0"
I20260927 10:46:20.509588 14212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b80217ddf067482793d1d3ab27c676d7"
format_stamp: "Formatted at 2026-09-27 10:46:20 on dist-test-slave-dzhw"
server_key: "bde0fd74426ba689bfe3c8fe68596407"
server_key_iv: "e1ca6d8ea164cc9a624f710a00c48a58"
server_key_version: "encryptionkey@0"
I20260927 10:46:20.513432 14212 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 10:46:20.515892 14227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:20.517148 14212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:46:20.517277 14212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "b80217ddf067482793d1d3ab27c676d7"
format_stamp: "Formatted at 2026-09-27 10:46:20 on dist-test-slave-dzhw"
server_key: "bde0fd74426ba689bfe3c8fe68596407"
server_key_iv: "e1ca6d8ea164cc9a624f710a00c48a58"
server_key_version: "encryptionkey@0"
I20260927 10:46:20.517391 14212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:20.529014 14212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:20.532178 14212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:20.532435 14212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:20.533056 14212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:46:20.533969 14212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:46:20.534017 14212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:20.534055 14212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:46:20.534070 14212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:20.543056 14212 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39729
I20260927 10:46:20.543081 14340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39729 every 8 connection(s)
I20260927 10:46:20.544185 14212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:46:20.549247 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 14212
I20260927 10:46:20.549391 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:46:20.549714 392 external_mini_cluster.cc:1428] Setting key 97cad75e68418ca395c9e2d442734e2d
Sep 27 10:46:20 dist-test-slave-dzhw krb5kdc[14105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:46:20.559634 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:20.546081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49891 (local address 127.0.98.62:41583)
0927 10:46:20.546240 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:20.546244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:20.547449 (+ 1205us) server_negotiation.cc:407] Connection header received
0927 10:46:20.548294 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:20.548297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:20.548344 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:20.548414 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:20.549938 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:20.550540 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:20.552154 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:20.552388 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:20.552471 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:20.555592 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:20.555596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:20.555598 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:20.555622 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:20.557359 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:20.557906 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:20.557912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:20.557916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:20.558027 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:20.558359 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:20.558366 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:20.558371 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:20.558653 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:20.558768 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:20.559284 (+ 516us) server_negotiation.cc:299] Negotiation successful
0927 10:46:20.559445 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 10:46:20.560469 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:20.546559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.1:49891)
0927 10:46:20.547242 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:20.547291 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:20.548099 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:20.548623 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:20.548630 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:20.549078 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:20.549786 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:20.549809 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:20.550656 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:20.550659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:20.552027 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:20.552042 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:20.552541 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:20.552547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:20.552798 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:20.553489 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:20.553506 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:20.555400 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:20.557500 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:20.557505 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:20.557518 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:20.557802 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:20.558124 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:20.558126 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:20.558127 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:20.558262 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:20.558761 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:20.558766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:20.559021 (+ 255us) client_negotiation.cc:769] Sending connection context
0927 10:46:20.559187 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 10:46:20.559456 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":115,"threads_started":1}
I20260927 10:46:20.561748 14341 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41583
I20260927 10:46:20.562008 14341 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:20.562587 14341 heartbeater.cc:511] Master 127.0.98.62:41583 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 10:46:20.564234 14150 ts_manager.cc:194] Registered new tserver with Master: b80217ddf067482793d1d3ab27c676d7 (127.0.98.1:39729)
I20260927 10:46:20.565411 14150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 10:46:20.581849 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41583
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:46:20.693394 14346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:20.693771 14346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:20.693854 14346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:20.698149 14346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:20.698218 14346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:20.698282 14346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:46:20.703644 14346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:20.704715 14346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:20.705680 14346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:20.715229 14352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:20.715302 14354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:20.715220 14351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:20.715487 14346 server_base.cc:1034] running on GCE node
I20260927 10:46:20.715931 14346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:20.716557 14346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:20.717706 14346 hybrid_clock.cc:648] HybridClock initialized: now 1790505980717693 us; error 26 us; skew 500 ppm
Sep 27 10:46:20 dist-test-slave-dzhw krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:20.720359 14346 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 10:46:20.721202 14346 webserver.cc:460] Webserver started at http://127.0.98.2:40557/ using document root <none> and password file <none>
I20260927 10:46:20.721771 14346 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:20.721830 14346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:20.722023 14346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:46:20.724261 14346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "50d424b2c709478ea641bdd86d39d613"
format_stamp: "Formatted at 2026-09-27 10:46:20 on dist-test-slave-dzhw"
server_key: "ad962da6a91e6657f5530acb01f81d59"
server_key_iv: "6953b5d0dcfe896534ec48074e0334fe"
server_key_version: "encryptionkey@0"
I20260927 10:46:20.724958 14346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "50d424b2c709478ea641bdd86d39d613"
format_stamp: "Formatted at 2026-09-27 10:46:20 on dist-test-slave-dzhw"
server_key: "ad962da6a91e6657f5530acb01f81d59"
server_key_iv: "6953b5d0dcfe896534ec48074e0334fe"
server_key_version: "encryptionkey@0"
I20260927 10:46:20.728749 14346 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:46:20.731127 14361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:20.732439 14346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:46:20.732578 14346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "50d424b2c709478ea641bdd86d39d613"
format_stamp: "Formatted at 2026-09-27 10:46:20 on dist-test-slave-dzhw"
server_key: "ad962da6a91e6657f5530acb01f81d59"
server_key_iv: "6953b5d0dcfe896534ec48074e0334fe"
server_key_version: "encryptionkey@0"
I20260927 10:46:20.732692 14346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:20.756125 14346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:20.759370 14346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:20.759582 14346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:20.760181 14346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:46:20.761022 14346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:46:20.761089 14346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:20.761157 14346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:46:20.761196 14346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:20.770191 14346 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39619
I20260927 10:46:20.770227 14474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39619 every 8 connection(s)
I20260927 10:46:20.771126 14346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:46:20.778766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 14346
I20260927 10:46:20.778914 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:46:20.779244 392 external_mini_cluster.cc:1428] Setting key 87bc078c83344c7ddf7920e12bd23773
Sep 27 10:46:20 dist-test-slave-dzhw krb5kdc[14105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:46:20.785105 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:20.773092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39493 (local address 127.0.98.62:41583)
0927 10:46:20.773238 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:20.773242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:20.774237 (+ 995us) server_negotiation.cc:407] Connection header received
0927 10:46:20.775212 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:20.775215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:20.775264 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:20.775347 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:20.776567 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:20.777100 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:20.778185 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:20.778406 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:20.778557 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:20.781509 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:20.781517 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:20.781519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:20.781541 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:20.783040 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:20.783594 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:20.783598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:20.783600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:20.783680 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:20.784006 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:20.784011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:20.784013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:20.784211 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:20.784316 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:20.784820 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 10:46:20.784943 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:46:20.786027 14477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:20.773430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.2:39493)
0927 10:46:20.774102 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:20.774149 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:20.774969 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:20.775551 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:20.775558 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:20.775915 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:20.776418 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:20.776433 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:20.777209 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:20.777211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:20.778076 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:20.778085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:20.778607 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:20.778612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:20.778817 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:20.779528 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:20.779549 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:20.781341 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:20.783147 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:20.783155 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:20.783172 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:20.783481 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:20.783769 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:20.783773 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:20.783775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:20.783916 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:20.784313 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:20.784320 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:20.784580 (+ 260us) client_negotiation.cc:769] Sending connection context
0927 10:46:20.784788 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 10:46:20.785154 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":113,"threads_started":1}
I20260927 10:46:20.787209 14475 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41583
I20260927 10:46:20.787479 14475 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:20.788126 14475 heartbeater.cc:511] Master 127.0.98.62:41583 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 10:46:20.789314 14150 ts_manager.cc:194] Registered new tserver with Master: 50d424b2c709478ea641bdd86d39d613 (127.0.98.2:39619)
I20260927 10:46:20.789937 14150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 10:46:20.809099 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41583
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:46:20.920284 14480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:20.920552 14480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:20.920641 14480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:20.924842 14480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:20.924913 14480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:20.925004 14480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:46:20.930389 14480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:20.931511 14480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:20.932525 14480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:20.946452 14485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:20.946568 14486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:20.946705 14488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:20.946789 14480 server_base.cc:1034] running on GCE node
I20260927 10:46:20.947247 14480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:20.947909 14480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:20.949079 14480 hybrid_clock.cc:648] HybridClock initialized: now 1790505980949036 us; error 55 us; skew 500 ppm
Sep 27 10:46:20 dist-test-slave-dzhw krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:20.951881 14480 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 10:46:20.952749 14480 webserver.cc:460] Webserver started at http://127.0.98.3:38365/ using document root <none> and password file <none>
I20260927 10:46:20.953317 14480 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:20.953377 14480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:20.953584 14480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:46:20.955683 14480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "f0026993090744a2973e7619c765f718"
format_stamp: "Formatted at 2026-09-27 10:46:20 on dist-test-slave-dzhw"
server_key: "6c1b7c4dd76015682138143ecd389d2c"
server_key_iv: "57813eb4f82ed73ee29d29d7345620b4"
server_key_version: "encryptionkey@0"
I20260927 10:46:20.956254 14480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f0026993090744a2973e7619c765f718"
format_stamp: "Formatted at 2026-09-27 10:46:20 on dist-test-slave-dzhw"
server_key: "6c1b7c4dd76015682138143ecd389d2c"
server_key_iv: "57813eb4f82ed73ee29d29d7345620b4"
server_key_version: "encryptionkey@0"
I20260927 10:46:20.959961 14480 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:46:20.962353 14495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:20.963656 14480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:46:20.963800 14480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "f0026993090744a2973e7619c765f718"
format_stamp: "Formatted at 2026-09-27 10:46:20 on dist-test-slave-dzhw"
server_key: "6c1b7c4dd76015682138143ecd389d2c"
server_key_iv: "57813eb4f82ed73ee29d29d7345620b4"
server_key_version: "encryptionkey@0"
I20260927 10:46:20.963909 14480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:20.997080 14480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:21.000156 14480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:21.000361 14480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:21.001113 14480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:46:21.002281 14480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:46:21.002380 14480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:21.002468 14480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:46:21.002523 14480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:21.012207 14480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42767
I20260927 10:46:21.012226 14608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42767 every 8 connection(s)
I20260927 10:46:21.013135 14480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:46:21.016142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 14480
I20260927 10:46:21.016273 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:46:21.016593 392 external_mini_cluster.cc:1428] Setting key 46315667fd4a3f420b123e14e712b706
Sep 27 10:46:21 dist-test-slave-dzhw krb5kdc[14105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:46:21.026809 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:21.014987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59977 (local address 127.0.98.62:41583)
0927 10:46:21.015125 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:21.015129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:21.016459 (+ 1330us) server_negotiation.cc:407] Connection header received
0927 10:46:21.017472 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:21.017476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:21.017532 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:21.017608 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:21.018824 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.019351 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.020461 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.020678 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.020793 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:21.023421 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:21.023424 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:21.023427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:21.023459 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:21.024808 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:21.025293 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:21.025298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:21.025300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:21.025379 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:21.025702 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:21.025706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:21.025708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:21.025887 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:21.026000 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:21.026497 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 10:46:21.026620 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:46:21.027587 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:21.015344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.3:59977)
0927 10:46:21.016299 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:21.016357 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:21.017263 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:21.017779 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:21.017787 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:21.018146 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:21.018693 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:21.018708 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.019469 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.019473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:21.020340 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:21.020349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.020831 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.020835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:21.021057 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:21.021650 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:21.021667 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:21.023259 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:21.024920 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:21.024926 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:21.024942 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:21.025190 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:21.025471 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:21.025474 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:21.025476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:21.025606 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:21.025996 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:21.026003 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:21.026254 (+ 251us) client_negotiation.cc:769] Sending connection context
0927 10:46:21.026453 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 10:46:21.026772 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":150,"threads_started":1}
I20260927 10:46:21.028676 14609 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41583
I20260927 10:46:21.028926 14609 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:21.029487 14609 heartbeater.cc:511] Master 127.0.98.62:41583 requested a full tablet report, sending...
I20260927 10:46:21.030550 14150 ts_manager.cc:194] Registered new tserver with Master: f0026993090744a2973e7619c765f718 (127.0.98.3:42767)
I20260927 10:46:21.031109 14150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59977
I20260927 10:46:21.042006 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:46:21.051023 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:21.043953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48310 (local address 127.0.98.62:41583)
0927 10:46:21.044089 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:21.044094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:21.044257 (+ 163us) server_negotiation.cc:407] Connection header received
0927 10:46:21.044356 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:21.044358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:21.044399 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:21.044465 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:21.045117 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.045632 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.046555 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.046780 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.046862 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:21.048215 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:21.048218 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:21.048220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:21.048243 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:21.049506 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:21.049898 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:21.049900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:21.049902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:21.049972 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:21.050194 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:21.050196 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:21.050197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:21.050405 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:21.050565 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:21.050735 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 10:46:21.050828 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:46:21.053972 14150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:46:21.057109 14150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:46:21.071869 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:21.066670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42767 (local address 127.0.0.1:40150)
0927 10:46:21.066911 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:21.066924 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:21.066999 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:21.067871 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:21.067873 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:21.067889 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:46:21.067967 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:21.067974 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.069437 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.069442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:21.070388 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:21.070398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.071333 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.071336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:21.071445 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:21.071480 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 10:46:21.071596 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 10:46:21.071716 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":34,"thread_start_us":100,"threads_started":1}
I20260927 10:46:21.071878 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:21.067134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40150 (local address 127.0.98.3:42767)
0927 10:46:21.067461 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:21.067472 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:21.067491 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:46:21.067555 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:21.067559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:21.067729 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:21.068102 (+ 373us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:46:21.068246 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.069311 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.070548 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.071210 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.071334 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:21.071501 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:21.071630 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 10:46:21.071715 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":120,"threads_started":1}
I20260927 10:46:21.072111 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:21.066000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39619 (local address 127.0.0.1:60558)
0927 10:46:21.066580 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:21.066592 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:21.066662 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:21.067681 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:21.067686 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:21.067707 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:46:21.067812 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:21.067818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.069064 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.069070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:21.070609 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:21.070620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.071689 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.071693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:21.071790 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:21.071811 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:46:21.071863 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 10:46:21.071941 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":91,"thread_start_us":111,"threads_started":1}
I20260927 10:46:21.072611 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:21.065893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39729 (local address 127.0.0.1:42846)
0927 10:46:21.066375 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:21.066444 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:21.066575 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:21.067461 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:21.067469 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:21.067493 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:46:21.067646 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:21.067652 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.069310 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.069313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:21.070609 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:21.070620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.071689 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.071693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:21.072333 (+ 640us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:21.072354 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:46:21.072428 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 10:46:21.072490 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":611,"thread_start_us":91,"threads_started":1}
I20260927 10:46:21.072649 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:21.066708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42846 (local address 127.0.98.1:39729)
0927 10:46:21.066998 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:21.067006 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:21.067029 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:46:21.067106 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:21.067111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:21.067294 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:21.067868 (+ 574us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:46:21.067989 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.069191 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.070758 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.071543 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.071736 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:21.071874 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:21.072461 (+ 587us) server_negotiation.cc:299] Negotiation successful
0927 10:46:21.072538 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":85,"threads_started":1}
I20260927 10:46:21.073184 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:21.066189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60558 (local address 127.0.98.2:39619)
0927 10:46:21.067235 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:21.067249 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:21.067274 (+ 25us) server_negotiation.cc:407] Connection header received
0927 10:46:21.067349 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:21.067354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:21.067529 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:21.067962 (+ 433us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:46:21.068120 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.068941 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.070757 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.071543 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.072796 (+ 1253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:21.072926 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:21.072970 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 10:46:21.073047 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":932,"thread_start_us":94,"threads_started":1}
I20260927 10:46:21.075363 14275 tablet_service.cc:1467] Processing CreateTablet for tablet 3d4317f3512643eaaf73ecc51d9ab1f2 (DEFAULT_TABLE table=test-table [id=dde3884183194037825b3c36751b73c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:46:21.075729 14543 tablet_service.cc:1467] Processing CreateTablet for tablet 3d4317f3512643eaaf73ecc51d9ab1f2 (DEFAULT_TABLE table=test-table [id=dde3884183194037825b3c36751b73c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:46:21.075729 14409 tablet_service.cc:1467] Processing CreateTablet for tablet 3d4317f3512643eaaf73ecc51d9ab1f2 (DEFAULT_TABLE table=test-table [id=dde3884183194037825b3c36751b73c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:46:21.076370 14275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d4317f3512643eaaf73ecc51d9ab1f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:46:21.076717 14409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d4317f3512643eaaf73ecc51d9ab1f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:46:21.076717 14543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d4317f3512643eaaf73ecc51d9ab1f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:46:21.082219 14626 tablet_bootstrap.cc:492] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7: Bootstrap starting.
I20260927 10:46:21.084508 14628 tablet_bootstrap.cc:492] T 3d4317f3512643eaaf73ecc51d9ab1f2 P f0026993090744a2973e7619c765f718: Bootstrap starting.
I20260927 10:46:21.084575 14626 tablet_bootstrap.cc:654] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7: Neither blocks nor log segments found. Creating new log.
I20260927 10:46:21.085302 14626 log.cc:824] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:21.085917 14627 tablet_bootstrap.cc:492] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613: Bootstrap starting.
I20260927 10:46:21.086727 14628 tablet_bootstrap.cc:654] T 3d4317f3512643eaaf73ecc51d9ab1f2 P f0026993090744a2973e7619c765f718: Neither blocks nor log segments found. Creating new log.
I20260927 10:46:21.087872 14628 log.cc:824] T 3d4317f3512643eaaf73ecc51d9ab1f2 P f0026993090744a2973e7619c765f718: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:21.088786 14627 tablet_bootstrap.cc:654] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613: Neither blocks nor log segments found. Creating new log.
I20260927 10:46:21.089463 14627 log.cc:824] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:21.089738 14628 tablet_bootstrap.cc:492] T 3d4317f3512643eaaf73ecc51d9ab1f2 P f0026993090744a2973e7619c765f718: No bootstrap required, opened a new log
I20260927 10:46:21.089960 14628 ts_tablet_manager.cc:1397] T 3d4317f3512643eaaf73ecc51d9ab1f2 P f0026993090744a2973e7619c765f718: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:46:21.091172 14626 tablet_bootstrap.cc:492] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7: No bootstrap required, opened a new log
I20260927 10:46:21.091346 14627 tablet_bootstrap.cc:492] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613: No bootstrap required, opened a new log
I20260927 10:46:21.091383 14626 ts_tablet_manager.cc:1397] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260927 10:46:21.091737 14627 ts_tablet_manager.cc:1397] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:46:21.093689 14628 raft_consensus.cc:348] T 3d4317f3512643eaaf73ecc51d9ab1f2 P f0026993090744a2973e7619c765f718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } }
I20260927 10:46:21.094033 14628 raft_consensus.cc:374] T 3d4317f3512643eaaf73ecc51d9ab1f2 P f0026993090744a2973e7619c765f718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:46:21.094085 14628 raft_consensus.cc:729] T 3d4317f3512643eaaf73ecc51d9ab1f2 P f0026993090744a2973e7619c765f718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0026993090744a2973e7619c765f718, State: Initialized, Role: FOLLOWER
I20260927 10:46:21.094592 14628 consensus_queue.cc:260] T 3d4317f3512643eaaf73ecc51d9ab1f2 P f0026993090744a2973e7619c765f718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } }
I20260927 10:46:21.095357 14609 heartbeater.cc:503] Master 127.0.98.62:41583 was elected leader, sending a full tablet report...
I20260927 10:46:21.095830 14627 raft_consensus.cc:348] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } }
I20260927 10:46:21.096109 14627 raft_consensus.cc:374] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:46:21.096168 14627 raft_consensus.cc:729] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50d424b2c709478ea641bdd86d39d613, State: Initialized, Role: FOLLOWER
I20260927 10:46:21.095988 14626 raft_consensus.cc:348] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } }
I20260927 10:46:21.096235 14626 raft_consensus.cc:374] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:46:21.096285 14626 raft_consensus.cc:729] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b80217ddf067482793d1d3ab27c676d7, State: Initialized, Role: FOLLOWER
I20260927 10:46:21.096606 14626 consensus_queue.cc:260] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } }
I20260927 10:46:21.096606 14627 consensus_queue.cc:260] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } }
I20260927 10:46:21.097245 14628 ts_tablet_manager.cc:1428] T 3d4317f3512643eaaf73ecc51d9ab1f2 P f0026993090744a2973e7619c765f718: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 10:46:21.097532 14626 ts_tablet_manager.cc:1428] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:46:21.097723 14627 ts_tablet_manager.cc:1428] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:46:21.097856 14341 heartbeater.cc:503] Master 127.0.98.62:41583 was elected leader, sending a full tablet report...
I20260927 10:46:21.098001 14475 heartbeater.cc:503] Master 127.0.98.62:41583 was elected leader, sending a full tablet report...
I20260927 10:46:21.239307 14633 raft_consensus.cc:482] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:46:21.239529 14633 raft_consensus.cc:504] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } }
I20260927 10:46:21.240720 14633 leader_election.cc:288] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b80217ddf067482793d1d3ab27c676d7 (127.0.98.1:39729), f0026993090744a2973e7619c765f718 (127.0.98.3:42767)
I20260927 10:46:21.244422 14477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:21.240962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39729 (local address 127.0.98.2:35849)
0927 10:46:21.241145 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:21.241161 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:21.241264 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:21.241526 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:21.241529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:21.241547 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:46:21.241668 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:21.241674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.242391 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.242396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:21.243290 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:21.243299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.244008 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.244011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:21.244119 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:21.244136 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 10:46:21.244223 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 10:46:21.244290 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 10:46:21.244422 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:21.241082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35849 (local address 127.0.98.1:39729)
0927 10:46:21.241268 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:21.241273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:21.241291 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:46:21.241358 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:21.241361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:21.241413 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:21.241505 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:46:21.241783 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.242283 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.243410 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.243890 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.243994 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:21.244056 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:21.244237 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 10:46:21.244305 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:46:21.244645 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:21.241164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50749 (local address 127.0.98.3:42767)
0927 10:46:21.241310 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:21.241315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:21.241594 (+ 279us) server_negotiation.cc:407] Connection header received
0927 10:46:21.241730 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:21.241733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:21.241780 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:21.241856 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:46:21.242111 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.242673 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.243685 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.244139 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.244249 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:21.244305 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:21.244470 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 10:46:21.244540 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 10:46:21.244660 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:21.240973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42767 (local address 127.0.98.2:50749)
0927 10:46:21.241502 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:21.241513 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:21.241623 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:21.241865 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:21.241868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:21.241883 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:46:21.242008 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:21.242013 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.242804 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.242807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:21.243570 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:21.243579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.244252 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.244255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:21.244348 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:21.244374 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 10:46:21.244456 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 10:46:21.244523 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":249,"thread_start_us":117,"threads_started":1}
I20260927 10:46:21.245131 14295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4317f3512643eaaf73ecc51d9ab1f2" candidate_uuid: "50d424b2c709478ea641bdd86d39d613" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b80217ddf067482793d1d3ab27c676d7" is_pre_election: true
I20260927 10:46:21.245332 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4317f3512643eaaf73ecc51d9ab1f2" candidate_uuid: "50d424b2c709478ea641bdd86d39d613" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0026993090744a2973e7619c765f718" is_pre_election: true
I20260927 10:46:21.245428 14295 raft_consensus.cc:2454] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50d424b2c709478ea641bdd86d39d613 in term 0.
I20260927 10:46:21.245600 14563 raft_consensus.cc:2454] T 3d4317f3512643eaaf73ecc51d9ab1f2 P f0026993090744a2973e7619c765f718 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50d424b2c709478ea641bdd86d39d613 in term 0.
I20260927 10:46:21.245927 14363 leader_election.cc:302] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50d424b2c709478ea641bdd86d39d613, b80217ddf067482793d1d3ab27c676d7; no voters:
I20260927 10:46:21.246207 14633 raft_consensus.cc:2781] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:46:21.246296 14633 raft_consensus.cc:482] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:46:21.246348 14633 raft_consensus.cc:3037] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:46:21.247413 14633 raft_consensus.cc:504] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } }
I20260927 10:46:21.247779 14633 leader_election.cc:288] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613 [CANDIDATE]: Term 1 election: Requested vote from peers b80217ddf067482793d1d3ab27c676d7 (127.0.98.1:39729), f0026993090744a2973e7619c765f718 (127.0.98.3:42767)
I20260927 10:46:21.248098 14295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4317f3512643eaaf73ecc51d9ab1f2" candidate_uuid: "50d424b2c709478ea641bdd86d39d613" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b80217ddf067482793d1d3ab27c676d7"
I20260927 10:46:21.248172 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4317f3512643eaaf73ecc51d9ab1f2" candidate_uuid: "50d424b2c709478ea641bdd86d39d613" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0026993090744a2973e7619c765f718"
I20260927 10:46:21.248209 14295 raft_consensus.cc:3037] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:46:21.248311 14563 raft_consensus.cc:3037] T 3d4317f3512643eaaf73ecc51d9ab1f2 P f0026993090744a2973e7619c765f718 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:46:21.249270 14295 raft_consensus.cc:2454] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50d424b2c709478ea641bdd86d39d613 in term 1.
I20260927 10:46:21.249377 14563 raft_consensus.cc:2454] T 3d4317f3512643eaaf73ecc51d9ab1f2 P f0026993090744a2973e7619c765f718 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50d424b2c709478ea641bdd86d39d613 in term 1.
I20260927 10:46:21.249660 14365 leader_election.cc:302] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50d424b2c709478ea641bdd86d39d613, f0026993090744a2973e7619c765f718; no voters:
I20260927 10:46:21.249840 14633 raft_consensus.cc:2781] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:46:21.250070 14633 raft_consensus.cc:686] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613 [term 1 LEADER]: Becoming Leader. State: Replica: 50d424b2c709478ea641bdd86d39d613, State: Running, Role: LEADER
I20260927 10:46:21.250382 14633 consensus_queue.cc:237] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } }
I20260927 10:46:21.253382 14150 catalog_manager.cc:5495] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50d424b2c709478ea641bdd86d39d613 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "50d424b2c709478ea641bdd86d39d613" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } health_report { overall_health: UNKNOWN } } }
W20260927 10:46:21.264052 14610 tablet.cc:2369] T 3d4317f3512643eaaf73ecc51d9ab1f2 P f0026993090744a2973e7619c765f718: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:46:21.272245 14476 tablet.cc:2369] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:46:21.273115 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:21.269173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.0.98.2:39619)
0927 10:46:21.269440 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:21.269443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:21.269453 (+ 10us) server_negotiation.cc:407] Connection header received
0927 10:46:21.269520 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:21.269522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:21.269574 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:21.269654 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:46:21.269943 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.270510 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.271628 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.271832 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.271904 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:21.272225 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:46:21.272697 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:46:21.272810 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:21.272887 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 10:46:21.272938 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192}
I20260927 10:46:21.279245 14563 raft_consensus.cc:1261] T 3d4317f3512643eaaf73ecc51d9ab1f2 P f0026993090744a2973e7619c765f718 [term 1 FOLLOWER]: Refusing update from remote peer 50d424b2c709478ea641bdd86d39d613: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:46:21.279245 14295 raft_consensus.cc:1261] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7 [term 1 FOLLOWER]: Refusing update from remote peer 50d424b2c709478ea641bdd86d39d613: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:46:21.280192 14636 consensus_queue.cc:1035] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:46:21.280545 14633 consensus_queue.cc:1035] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:46:21.290081 14640 mvcc.cc:205] Tried to move back new op lower bound from 7333912499312263168 to 7333912499204325376. Current Snapshot: MvccSnapshot[applied={T|T < 7333912499312263168}]
I20260927 10:46:21.300036 14149 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:46:21.300207 14149 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:46:21.302615 14149 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb: Sending DeleteTablet for 3 replicas of tablet 3d4317f3512643eaaf73ecc51d9ab1f2
I20260927 10:46:21.303406 14409 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d4317f3512643eaaf73ecc51d9ab1f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:46:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60558
I20260927 10:46:21.303401 14543 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d4317f3512643eaaf73ecc51d9ab1f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:46:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40150
I20260927 10:46:21.303428 14275 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d4317f3512643eaaf73ecc51d9ab1f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:46:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42846
I20260927 10:46:21.303930 14650 tablet_replica.cc:330] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7: stopping tablet replica
I20260927 10:46:21.304175 14648 tablet_replica.cc:330] T 3d4317f3512643eaaf73ecc51d9ab1f2 P f0026993090744a2973e7619c765f718: stopping tablet replica
I20260927 10:46:21.304278 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 14212
I20260927 10:46:21.304291 14650 raft_consensus.cc:2229] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:46:21.304435 14649 tablet_replica.cc:330] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613: stopping tablet replica
I20260927 10:46:21.304653 14650 raft_consensus.cc:2258] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:46:21.304970 14649 raft_consensus.cc:2229] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:46:21.305382 14649 raft_consensus.cc:2258] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:46:21.305688 14648 raft_consensus.cc:2229] T 3d4317f3512643eaaf73ecc51d9ab1f2 P f0026993090744a2973e7619c765f718 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:46:21.305959 14648 raft_consensus.cc:2258] T 3d4317f3512643eaaf73ecc51d9ab1f2 P f0026993090744a2973e7619c765f718 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:46:21.307387 14649 ts_tablet_manager.cc:1905] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:46:21.308746 14648 ts_tablet_manager.cc:1905] T 3d4317f3512643eaaf73ecc51d9ab1f2 P f0026993090744a2973e7619c765f718: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:46:21.310314 14649 ts_tablet_manager.cc:1918] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:46:21.310498 14649 log.cc:1196] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/wal/wals/3d4317f3512643eaaf73ecc51d9ab1f2
I20260927 10:46:21.310899 14649 ts_tablet_manager.cc:1939] T 3d4317f3512643eaaf73ecc51d9ab1f2 P 50d424b2c709478ea641bdd86d39d613: Deleting consensus metadata
I20260927 10:46:21.312083 14648 ts_tablet_manager.cc:1918] T 3d4317f3512643eaaf73ecc51d9ab1f2 P f0026993090744a2973e7619c765f718: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:46:21.312184 14648 log.cc:1196] T 3d4317f3512643eaaf73ecc51d9ab1f2 P f0026993090744a2973e7619c765f718: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/3d4317f3512643eaaf73ecc51d9ab1f2
I20260927 10:46:21.312355 14136 catalog_manager.cc:4841] TS 50d424b2c709478ea641bdd86d39d613 (127.0.98.2:39619): tablet 3d4317f3512643eaaf73ecc51d9ab1f2 (table test-table [id=dde3884183194037825b3c36751b73c7]) successfully deleted
I20260927 10:46:21.312522 14648 ts_tablet_manager.cc:1939] T 3d4317f3512643eaaf73ecc51d9ab1f2 P f0026993090744a2973e7619c765f718: Deleting consensus metadata
W20260927 10:46:21.313220 14135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39729 (error 108)
I20260927 10:46:21.313256 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 14346
I20260927 10:46:21.314604 14137 catalog_manager.cc:4841] TS f0026993090744a2973e7619c765f718 (127.0.98.3:42767): tablet 3d4317f3512643eaaf73ecc51d9ab1f2 (table test-table [id=dde3884183194037825b3c36751b73c7]) successfully deleted
I20260927 10:46:21.315546 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:21.315175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39729 (local address 127.0.0.1:42858)
0927 10:46:21.315344 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:21.315429 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 10:46:21.315716 14135 catalog_manager.cc:4584] TS b80217ddf067482793d1d3ab27c676d7 (127.0.98.1:39729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d4317f3512643eaaf73ecc51d9ab1f2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39729: connect: Connection refused (error 111)
I20260927 10:46:21.319331 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 14480
I20260927 10:46:21.324884 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 14118
I20260927 10:46:21.331673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41583
--webserver_interface=127.0.98.62
--webserver_port=38861
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:46:21.443943 14652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:21.444226 14652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:21.444293 14652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:21.448365 14652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:46:21.448429 14652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:21.448449 14652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:21.448467 14652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:46:21.448508 14652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:46:21.453922 14652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:21.455149 14652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:21.456200 14652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:21.470168 14661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:21.470562 14652 server_base.cc:1034] running on GCE node
W20260927 10:46:21.470181 14658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:21.470173 14659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:21.471354 14652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:21.472838 14652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:21.474144 14652 hybrid_clock.cc:648] HybridClock initialized: now 1790505981474112 us; error 47 us; skew 500 ppm
Sep 27 10:46:21 dist-test-slave-dzhw krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:21.477450 14652 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:46:21.478586 14652 webserver.cc:460] Webserver started at http://127.0.98.62:38861/ using document root <none> and password file <none>
I20260927 10:46:21.479197 14652 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:21.479302 14652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:21.483736 14652 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:46:21.486266 14668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:21.487747 14652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:46:21.487900 14652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "74aa3fb48ada44db96baf018b173eefb"
format_stamp: "Formatted at 2026-09-27 10:46:20 on dist-test-slave-dzhw"
server_key: "8cf99d10a6d612f121b97bfea5bd45ed"
server_key_iv: "ed557d852b6de1e1fed9a82bd262e691"
server_key_version: "encryptionkey@0"
I20260927 10:46:21.488544 14652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:21.516875 14652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:21.520267 14652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:21.520479 14652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:21.527479 14652 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41583
I20260927 10:46:21.527786 14720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41583 every 8 connection(s)
I20260927 10:46:21.528586 14652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:46:21.528863 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 14652
I20260927 10:46:21.529369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39729
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41583
--webserver_port=35261
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:46:21.533190 14721 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:46:21.534945 14721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb: Bootstrap starting.
I20260927 10:46:21.551669 14721 log.cc:824] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:21.559974 14721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:46:21.560384 14721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb: Bootstrap complete.
I20260927 10:46:21.564479 14721 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74aa3fb48ada44db96baf018b173eefb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41583 } }
I20260927 10:46:21.564893 14721 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74aa3fb48ada44db96baf018b173eefb, State: Initialized, Role: FOLLOWER
I20260927 10:46:21.565367 14721 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74aa3fb48ada44db96baf018b173eefb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41583 } }
I20260927 10:46:21.565506 14721 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:46:21.565588 14721 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:46:21.565702 14721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:46:21.567695 14721 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74aa3fb48ada44db96baf018b173eefb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41583 } }
I20260927 10:46:21.568027 14721 leader_election.cc:302] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74aa3fb48ada44db96baf018b173eefb; no voters:
I20260927 10:46:21.568364 14721 leader_election.cc:288] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 10:46:21.568454 14725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [term 2 FOLLOWER]: Leader election won for term 2
I20260927 10:46:21.568691 14725 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [term 2 LEADER]: Becoming Leader. State: Replica: 74aa3fb48ada44db96baf018b173eefb, State: Running, Role: LEADER
I20260927 10:46:21.568984 14725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74aa3fb48ada44db96baf018b173eefb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41583 } }
I20260927 10:46:21.569614 14721 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:46:21.569902 14725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74aa3fb48ada44db96baf018b173eefb. Latest consensus state: current_term: 2 leader_uuid: "74aa3fb48ada44db96baf018b173eefb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74aa3fb48ada44db96baf018b173eefb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41583 } } }
I20260927 10:46:21.570021 14725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [sys.catalog]: This master's current role is: LEADER
I20260927 10:46:21.570660 14726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "74aa3fb48ada44db96baf018b173eefb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74aa3fb48ada44db96baf018b173eefb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41583 } } }
I20260927 10:46:21.570765 14726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [sys.catalog]: This master's current role is: LEADER
I20260927 10:46:21.571504 14729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:46:21.576169 14729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:46:21.576812 14729 catalog_manager.cc:1259] Loaded cluster ID: 36a62e5caf1643839c0213c6011b43b2
I20260927 10:46:21.576875 14729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:46:21.578253 14729 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 10:46:21.583101 14729 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 10:46:21.588783 14729 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:46:21.589578 14729 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb: Loaded TSK: 0
I20260927 10:46:21.590796 14729 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:46:21.644325 14723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:21.644673 14723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:21.644757 14723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:21.648862 14723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:21.648959 14723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:21.649060 14723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:46:21.654407 14723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35261
--tserver_master_addrs=127.0.98.62:41583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:21.655622 14723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:21.656659 14723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:21.666072 14751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:21.666250 14748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:21.666314 14723 server_base.cc:1034] running on GCE node
W20260927 10:46:21.666071 14749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:21.666747 14723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:21.667419 14723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:21.668622 14723 hybrid_clock.cc:648] HybridClock initialized: now 1790505981668597 us; error 42 us; skew 500 ppm
Sep 27 10:46:21 dist-test-slave-dzhw krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:21.671926 14723 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 10:46:21.672957 14723 webserver.cc:460] Webserver started at http://127.0.98.1:35261/ using document root <none> and password file <none>
I20260927 10:46:21.673540 14723 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:21.673643 14723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:21.677781 14723 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:46:21.680161 14758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:21.681468 14723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:46:21.681568 14723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "b80217ddf067482793d1d3ab27c676d7"
format_stamp: "Formatted at 2026-09-27 10:46:20 on dist-test-slave-dzhw"
server_key: "bde0fd74426ba689bfe3c8fe68596407"
server_key_iv: "e1ca6d8ea164cc9a624f710a00c48a58"
server_key_version: "encryptionkey@0"
I20260927 10:46:21.682144 14723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:21.693126 14723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:21.696754 14723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:21.696961 14723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:21.697598 14723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:46:21.698812 14765 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 10:46:21.702893 14723 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 10:46:21.702957 14723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260927 10:46:21.703030 14723 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 10:46:21.704697 14723 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 10:46:21.704757 14723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 10:46:21.705001 14765 tablet_bootstrap.cc:492] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7: Bootstrap starting.
I20260927 10:46:21.715706 14723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39729
I20260927 10:46:21.715734 14872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39729 every 8 connection(s)
I20260927 10:46:21.716979 14723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:46:21.725935 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 14723
I20260927 10:46:21.726260 14765 log.cc:824] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:21.726490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39619
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41583
--webserver_port=40557
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:46:21 dist-test-slave-dzhw krb5kdc[14105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:46:21.734481 14765 tablet_bootstrap.cc:492] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:46:21.734863 14765 tablet_bootstrap.cc:492] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7: Bootstrap complete.
I20260927 10:46:21.736243 14765 ts_tablet_manager.cc:1397] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7: Time spent bootstrapping tablet: real 0.031s user 0.013s sys 0.016s
I20260927 10:46:21.736865 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:21.719321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.1:39391)
0927 10:46:21.720095 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:21.720149 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:21.721119 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:21.722575 (+ 1456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:21.722590 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:21.723121 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:21.723834 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:21.723865 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.725137 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.725141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:21.726486 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:21.726497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.727468 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.727471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:21.727661 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:21.728393 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:21.728418 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:21.730352 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:21.733178 (+ 2826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:21.733184 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:21.733195 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:21.733539 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:21.733982 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:21.733985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:21.733987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:21.734148 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:21.734999 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:21.735005 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:21.735317 (+ 312us) client_negotiation.cc:769] Sending connection context
0927 10:46:21.735505 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 10:46:21.735852 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":160,"threads_started":1}
I20260927 10:46:21.736965 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:21.719326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39391 (local address 127.0.98.62:41583)
0927 10:46:21.719793 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:21.719814 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:21.721419 (+ 1605us) server_negotiation.cc:407] Connection header received
0927 10:46:21.721688 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:21.721700 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:21.722050 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:21.722382 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:21.724039 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.724991 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.726637 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.727315 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.727502 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:21.730643 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:21.730653 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:21.730672 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:21.730715 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:21.733037 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:21.733683 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:21.733692 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:21.733700 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:21.733868 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:21.734269 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:21.734274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:21.734277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:21.734837 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:21.734987 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:21.735688 (+ 701us) server_negotiation.cc:299] Negotiation successful
0927 10:46:21.736023 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":110,"threads_started":1}
I20260927 10:46:21.739080 14765 raft_consensus.cc:348] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } }
I20260927 10:46:21.739537 14765 raft_consensus.cc:729] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b80217ddf067482793d1d3ab27c676d7, State: Initialized, Role: FOLLOWER
I20260927 10:46:21.739897 14873 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41583
I20260927 10:46:21.740118 14873 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:21.740095 14765 consensus_queue.cc:260] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } }
I20260927 10:46:21.740665 14873 heartbeater.cc:511] Master 127.0.98.62:41583 requested a full tablet report, sending...
I20260927 10:46:21.741086 14765 ts_tablet_manager.cc:1428] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:46:21.743208 14685 ts_manager.cc:194] Registered new tserver with Master: b80217ddf067482793d1d3ab27c676d7 (127.0.98.1:39729)
I20260927 10:46:21.746508 14685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39391
I20260927 10:46:21.749099 14873 heartbeater.cc:503] Master 127.0.98.62:41583 was elected leader, sending a full tablet report...
Sep 27 10:46:21 dist-test-slave-dzhw krb5kdc[14105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 10:46:21.757176 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:21.745967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39729 (local address 127.0.0.1:42864)
0927 10:46:21.746448 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:21.746491 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:21.746624 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:21.747899 (+ 1275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:21.747906 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:21.748353 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:21.748491 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:21.748498 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.749700 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.749705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:21.750780 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:21.750792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.751352 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.751369 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:21.751471 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:21.752005 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:21.752019 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:21.753459 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:21.755305 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:21.755324 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:21.755330 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:21.755570 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:21.755888 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:21.755891 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:21.755893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:21.755980 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:21.756469 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:21.756475 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:21.756602 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 10:46:21.756774 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 10:46:21.757003 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":155,"threads_started":1}
I20260927 10:46:21.757176 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:21.746078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42864 (local address 127.0.98.1:39729)
0927 10:46:21.747425 (+ 1347us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:21.747434 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:21.747452 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:46:21.747541 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:21.747546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:21.747735 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:21.747873 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:21.748618 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.749583 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.750885 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.751249 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.751338 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:21.753606 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:21.753621 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:21.753630 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:21.753653 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:21.755172 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:21.755694 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:21.755700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:21.755702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:21.755783 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:21.756084 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:21.756087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:21.756089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:21.756328 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:21.756477 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:21.756855 (+ 378us) server_negotiation.cc:299] Negotiation successful
0927 10:46:21.756999 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1261,"thread_start_us":87,"threads_started":1}
I20260927 10:46:21.758580 14807 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d4317f3512643eaaf73ecc51d9ab1f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:46:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42864
I20260927 10:46:21.759434 14882 tablet_replica.cc:330] stopping tablet replica
I20260927 10:46:21.759750 14882 raft_consensus.cc:2229] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:46:21.760000 14882 raft_consensus.cc:2258] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:46:21.761072 14882 ts_tablet_manager.cc:1905] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:46:21.765704 14882 ts_tablet_manager.cc:1918] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:46:21.765817 14882 log.cc:1196] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/wal/wals/3d4317f3512643eaaf73ecc51d9ab1f2
I20260927 10:46:21.766116 14882 ts_tablet_manager.cc:1939] T 3d4317f3512643eaaf73ecc51d9ab1f2 P b80217ddf067482793d1d3ab27c676d7: Deleting consensus metadata
I20260927 10:46:21.768146 14670 catalog_manager.cc:4841] TS b80217ddf067482793d1d3ab27c676d7 (127.0.98.1:39729): tablet 3d4317f3512643eaaf73ecc51d9ab1f2 (table test-table [id=dde3884183194037825b3c36751b73c7]) successfully deleted
W20260927 10:46:21.858661 14878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:21.859114 14878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:21.859200 14878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:21.868216 14878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:21.868316 14878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:21.868453 14878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:46:21.876101 14878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39619
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40557
--tserver_master_addrs=127.0.98.62:41583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:21.877194 14878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:21.878161 14878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:21.888113 14890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:21.888142 14887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:21.888230 14878 server_base.cc:1034] running on GCE node
W20260927 10:46:21.888142 14888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:21.888862 14878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:21.889541 14878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:21.890746 14878 hybrid_clock.cc:648] HybridClock initialized: now 1790505981890721 us; error 45 us; skew 500 ppm
Sep 27 10:46:21 dist-test-slave-dzhw krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:21.894456 14878 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 10:46:21.895398 14878 webserver.cc:460] Webserver started at http://127.0.98.2:40557/ using document root <none> and password file <none>
I20260927 10:46:21.896004 14878 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:21.896080 14878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:21.899961 14878 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:46:21.902074 14897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:21.903354 14878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:46:21.903499 14878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "50d424b2c709478ea641bdd86d39d613"
format_stamp: "Formatted at 2026-09-27 10:46:20 on dist-test-slave-dzhw"
server_key: "ad962da6a91e6657f5530acb01f81d59"
server_key_iv: "6953b5d0dcfe896534ec48074e0334fe"
server_key_version: "encryptionkey@0"
I20260927 10:46:21.904052 14878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:21.915676 14878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:21.918866 14878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:21.919070 14878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:21.919661 14878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:46:21.920485 14878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:46:21.920559 14878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:21.920648 14878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:46:21.920696 14878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:21.930053 14878 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39619
I20260927 10:46:21.930078 15010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39619 every 8 connection(s)
I20260927 10:46:21.931172 14878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:46:21.933929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 14878
I20260927 10:46:21.934456 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42767
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41583
--webserver_port=38365
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:46:21 dist-test-slave-dzhw krb5kdc[14105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:46:21.946593 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:21.933035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40997 (local address 127.0.98.62:41583)
0927 10:46:21.933191 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:21.933195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:21.934345 (+ 1150us) server_negotiation.cc:407] Connection header received
0927 10:46:21.935785 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:21.935791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:21.935865 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:21.935979 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:21.937133 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.938045 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.939415 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.939866 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.939999 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:21.942528 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:21.942534 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:21.942538 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:21.942581 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:21.944449 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:21.944959 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:21.944964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:21.944968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:21.945068 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:21.945395 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:21.945400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:21.945405 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:21.945641 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:21.945738 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:21.946247 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 10:46:21.946386 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:46:21.947415 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:21.933424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.2:40997)
0927 10:46:21.934171 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:21.934231 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:21.935512 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:21.936156 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:21.936162 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:21.936511 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:21.936984 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:21.936997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.938194 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.938196 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:21.939290 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:21.939299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:21.940022 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:21.940025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:21.940188 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:21.940812 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:21.940829 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:21.942293 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:21.944573 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:21.944578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:21.944592 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:21.944857 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:21.945169 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:21.945171 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:21.945172 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:21.945302 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:21.945746 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:21.945751 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:21.946005 (+ 254us) client_negotiation.cc:769] Sending connection context
0927 10:46:21.946178 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 10:46:21.946503 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":104,"threads_started":1}
I20260927 10:46:21.948593 15011 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41583
I20260927 10:46:21.948823 15011 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:21.949314 15011 heartbeater.cc:511] Master 127.0.98.62:41583 requested a full tablet report, sending...
I20260927 10:46:21.950318 14685 ts_manager.cc:194] Registered new tserver with Master: 50d424b2c709478ea641bdd86d39d613 (127.0.98.2:39619)
I20260927 10:46:21.950909 14685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40997
W20260927 10:46:22.048285 15014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:22.048561 15014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:22.048605 15014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:22.052763 15014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:22.052826 15014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:22.052935 15014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:46:22.058187 15014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38365
--tserver_master_addrs=127.0.98.62:41583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:22.059281 15014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:22.060300 15014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:22.070637 15019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:22.070652 15022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:22.070633 15020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:22.070901 15014 server_base.cc:1034] running on GCE node
I20260927 10:46:22.071383 15014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:22.072052 15014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:22.073216 15014 hybrid_clock.cc:648] HybridClock initialized: now 1790505982073198 us; error 33 us; skew 500 ppm
Sep 27 10:46:22 dist-test-slave-dzhw krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:22.075924 15014 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 10:46:22.076828 15014 webserver.cc:460] Webserver started at http://127.0.98.3:38365/ using document root <none> and password file <none>
I20260927 10:46:22.077371 15014 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:22.077461 15014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:22.081281 15014 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:46:22.083266 15029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:22.084522 15014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:46:22.084658 15014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "f0026993090744a2973e7619c765f718"
format_stamp: "Formatted at 2026-09-27 10:46:20 on dist-test-slave-dzhw"
server_key: "6c1b7c4dd76015682138143ecd389d2c"
server_key_iv: "57813eb4f82ed73ee29d29d7345620b4"
server_key_version: "encryptionkey@0"
I20260927 10:46:22.085193 15014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:22.095872 15014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:22.098862 15014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:22.099063 15014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:22.099655 15014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:46:22.100451 15014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:46:22.100525 15014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:22.100610 15014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:46:22.100656 15014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:22.109609 15014 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42767
I20260927 10:46:22.109617 15142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42767 every 8 connection(s)
I20260927 10:46:22.110801 15014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:46:22.111516 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 15014
I20260927 10:46:22.120508 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:22.112261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48312 (local address 127.0.98.62:41583)
0927 10:46:22.112395 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:22.112400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:22.112685 (+ 285us) server_negotiation.cc:407] Connection header received
0927 10:46:22.112845 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:22.112848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:22.112890 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:22.112955 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:22.113914 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.114734 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.115698 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.116036 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.116143 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:22.117461 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:22.117465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:22.117468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:22.117492 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:22.118938 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:22.119355 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:22.119360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:22.119362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:22.119441 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:22.119701 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:22.119706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:22.119708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:22.119874 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:22.120074 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:22.120222 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 10:46:22.120337 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 10:46:22 dist-test-slave-dzhw krb5kdc[14105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:46:22.124827 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:22.112631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59145 (local address 127.0.98.62:41583)
0927 10:46:22.113103 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:22.113106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:22.114244 (+ 1138us) server_negotiation.cc:407] Connection header received
0927 10:46:22.115300 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:22.115304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:22.115351 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:22.115445 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:22.116756 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.117249 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.118674 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.118911 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.119015 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:22.121592 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:22.121595 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:22.121597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:22.121623 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:22.123079 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:22.123569 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:22.123571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:22.123572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:22.123642 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:22.123934 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:22.123936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:22.123937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:22.124088 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:22.124162 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:22.124602 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 10:46:22.124701 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":74,"threads_started":1}
I20260927 10:46:22.125716 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:22.113232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.3:59145)
0927 10:46:22.114060 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:22.114115 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:22.115114 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:22.115609 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:22.115615 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:22.116090 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:22.116624 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:22.116637 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.117369 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.117373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:22.118554 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:22.118564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.119056 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.119061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:22.119232 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:22.119751 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:22.119767 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:22.121433 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:22.123197 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:22.123202 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:22.123214 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:22.123475 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:22.123724 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:22.123727 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:22.123728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:22.123858 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:22.124172 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:22.124177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:22.124405 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 10:46:22.124573 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 10:46:22.124820 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":172,"threads_started":1}
I20260927 10:46:22.126880 15143 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41583
I20260927 10:46:22.127110 15143 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:22.127573 15143 heartbeater.cc:511] Master 127.0.98.62:41583 requested a full tablet report, sending...
I20260927 10:46:22.128615 14685 ts_manager.cc:194] Registered new tserver with Master: f0026993090744a2973e7619c765f718 (127.0.98.3:42767)
I20260927 10:46:22.129199 14685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59145
I20260927 10:46:22.134191 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:46:22.138231 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:22.134861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.0.98.62:41583)
0927 10:46:22.134992 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:22.134995 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:22.135190 (+ 195us) server_negotiation.cc:407] Connection header received
0927 10:46:22.135327 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:22.135329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:22.135427 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:22.135515 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:46:22.135738 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.136237 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.137022 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.137242 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.137317 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:22.137565 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:46:22.137942 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:46:22.138027 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:22.138075 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 10:46:22.138126 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:46:22.138896 14685 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:46:22.142155 14685 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:46:22.150017 14807 tablet_service.cc:1467] Processing CreateTablet for tablet b918621d4835470ab29b8e6fa6c32412 (DEFAULT_TABLE table=test-table [id=8004cac10d8545f8bb5c8cb98fd377d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:46:22.150704 14807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b918621d4835470ab29b8e6fa6c32412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:46:22.153358 14765 tablet_bootstrap.cc:492] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7: Bootstrap starting.
I20260927 10:46:22.154266 14765 tablet_bootstrap.cc:654] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7: Neither blocks nor log segments found. Creating new log.
I20260927 10:46:22.154932 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:22.148994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42767 (local address 127.0.0.1:40166)
0927 10:46:22.149122 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:22.149139 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:22.149236 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:22.150528 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:22.150531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:22.150551 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:46:22.150664 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:22.150670 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.152022 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.152029 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:22.153188 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:22.153208 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.154506 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.154510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:22.154605 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:22.154645 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 10:46:22.154701 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 10:46:22.154784 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 10:46:22.155092 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:22.149270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39619 (local address 127.0.0.1:60564)
0927 10:46:22.149581 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:22.149597 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:22.149683 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:22.150709 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:22.150712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:22.150757 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:46:22.150851 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:22.150857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.152411 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.152415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:22.153485 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:22.153493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.154743 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.154746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:22.154820 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:22.154839 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 10:46:22.154911 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 10:46:22.154980 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":100,"threads_started":1}
I20260927 10:46:22.155314 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:22.149700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60564 (local address 127.0.98.2:39619)
0927 10:46:22.149970 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:22.149987 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:22.150011 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:46:22.150092 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:22.150099 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:22.150338 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:22.150549 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:46:22.150975 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.152261 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.153612 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.154372 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.154527 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:22.154679 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:22.154969 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 10:46:22.155074 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":88,"threads_started":1}
I20260927 10:46:22.155736 15152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:22.149186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40166 (local address 127.0.98.3:42767)
0927 10:46:22.150079 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:22.150089 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:22.150106 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:46:22.150168 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:22.150172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:22.150337 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:22.150549 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:46:22.150790 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.151893 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.153336 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.154332 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.155325 (+ 993us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:22.155456 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:22.155503 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 10:46:22.155595 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804,"thread_start_us":150,"threads_started":1}
I20260927 10:46:22.158471 14765 tablet_bootstrap.cc:492] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7: No bootstrap required, opened a new log
I20260927 10:46:22.158572 14765 ts_tablet_manager.cc:1397] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 10:46:22.158133 15077 tablet_service.cc:1467] Processing CreateTablet for tablet b918621d4835470ab29b8e6fa6c32412 (DEFAULT_TABLE table=test-table [id=8004cac10d8545f8bb5c8cb98fd377d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:46:22.158136 14945 tablet_service.cc:1467] Processing CreateTablet for tablet b918621d4835470ab29b8e6fa6c32412 (DEFAULT_TABLE table=test-table [id=8004cac10d8545f8bb5c8cb98fd377d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:46:22.159062 14765 raft_consensus.cc:348] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } }
I20260927 10:46:22.159194 14765 raft_consensus.cc:374] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:46:22.159235 14765 raft_consensus.cc:729] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b80217ddf067482793d1d3ab27c676d7, State: Initialized, Role: FOLLOWER
I20260927 10:46:22.159307 15077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b918621d4835470ab29b8e6fa6c32412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:46:22.159312 14945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b918621d4835470ab29b8e6fa6c32412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:46:22.159377 14765 consensus_queue.cc:260] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } }
I20260927 10:46:22.159855 14765 ts_tablet_manager.cc:1428] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:46:22.166347 15156 tablet_bootstrap.cc:492] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613: Bootstrap starting.
I20260927 10:46:22.168910 15156 tablet_bootstrap.cc:654] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613: Neither blocks nor log segments found. Creating new log.
I20260927 10:46:22.169492 15157 tablet_bootstrap.cc:492] T b918621d4835470ab29b8e6fa6c32412 P f0026993090744a2973e7619c765f718: Bootstrap starting.
I20260927 10:46:22.169718 15156 log.cc:824] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:22.171571 15157 tablet_bootstrap.cc:654] T b918621d4835470ab29b8e6fa6c32412 P f0026993090744a2973e7619c765f718: Neither blocks nor log segments found. Creating new log.
I20260927 10:46:22.171787 15156 tablet_bootstrap.cc:492] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613: No bootstrap required, opened a new log
I20260927 10:46:22.171985 15156 ts_tablet_manager.cc:1397] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:46:22.172634 15157 log.cc:824] T b918621d4835470ab29b8e6fa6c32412 P f0026993090744a2973e7619c765f718: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:22.174387 15157 tablet_bootstrap.cc:492] T b918621d4835470ab29b8e6fa6c32412 P f0026993090744a2973e7619c765f718: No bootstrap required, opened a new log
I20260927 10:46:22.174593 15157 ts_tablet_manager.cc:1397] T b918621d4835470ab29b8e6fa6c32412 P f0026993090744a2973e7619c765f718: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:46:22.175366 15156 raft_consensus.cc:348] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } }
I20260927 10:46:22.175612 15156 raft_consensus.cc:374] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:46:22.175676 15156 raft_consensus.cc:729] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50d424b2c709478ea641bdd86d39d613, State: Initialized, Role: FOLLOWER
I20260927 10:46:22.176072 15156 consensus_queue.cc:260] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } }
I20260927 10:46:22.176689 15011 heartbeater.cc:503] Master 127.0.98.62:41583 was elected leader, sending a full tablet report...
I20260927 10:46:22.177057 15156 ts_tablet_manager.cc:1428] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:46:22.177500 15157 raft_consensus.cc:348] T b918621d4835470ab29b8e6fa6c32412 P f0026993090744a2973e7619c765f718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } }
I20260927 10:46:22.177789 15157 raft_consensus.cc:374] T b918621d4835470ab29b8e6fa6c32412 P f0026993090744a2973e7619c765f718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:46:22.177839 15157 raft_consensus.cc:729] T b918621d4835470ab29b8e6fa6c32412 P f0026993090744a2973e7619c765f718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0026993090744a2973e7619c765f718, State: Initialized, Role: FOLLOWER
I20260927 10:46:22.178263 15157 consensus_queue.cc:260] T b918621d4835470ab29b8e6fa6c32412 P f0026993090744a2973e7619c765f718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } }
I20260927 10:46:22.179287 15157 ts_tablet_manager.cc:1428] T b918621d4835470ab29b8e6fa6c32412 P f0026993090744a2973e7619c765f718: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:46:22.179652 15143 heartbeater.cc:503] Master 127.0.98.62:41583 was elected leader, sending a full tablet report...
W20260927 10:46:22.182106 15012 tablet.cc:2369] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:46:22.218456 14874 tablet.cc:2369] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:46:22.361802 15144 tablet.cc:2369] T b918621d4835470ab29b8e6fa6c32412 P f0026993090744a2973e7619c765f718: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:46:22.408998 15160 raft_consensus.cc:482] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:46:22.409257 15160 raft_consensus.cc:504] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } }
I20260927 10:46:22.410490 15160 leader_election.cc:288] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0026993090744a2973e7619c765f718 (127.0.98.3:42767), b80217ddf067482793d1d3ab27c676d7 (127.0.98.1:39729)
I20260927 10:46:22.414597 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:22.410788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42767 (local address 127.0.98.2:35585)
0927 10:46:22.410958 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:22.410976 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:22.411121 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:22.411404 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:22.411407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:22.411425 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:46:22.411560 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:22.411570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.412386 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.412390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:22.413292 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:22.413302 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.414098 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.414102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:22.414213 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:22.414240 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:46:22.414326 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 10:46:22.414443 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":2}
I20260927 10:46:22.414623 15152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:22.411014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35585 (local address 127.0.98.3:42767)
0927 10:46:22.411156 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:22.411160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:22.411177 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:46:22.411242 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:22.411247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:22.411302 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:22.411375 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:46:22.411676 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.412250 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.413430 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.413942 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.414021 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:22.414078 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:22.414379 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 10:46:22.414475 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:46:22.414858 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:22.410788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39729 (local address 127.0.98.2:50167)
0927 10:46:22.411182 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:22.411196 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:22.411298 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:22.411642 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:22.411647 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:22.411665 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:46:22.411769 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:22.411776 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.412654 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.412657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:22.413573 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:22.413583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.414452 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.414456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:22.414550 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:22.414573 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:46:22.414640 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 10:46:22.414713 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":133,"thread_start_us":75,"threads_started":1}
I20260927 10:46:22.414906 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:22.410914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50167 (local address 127.0.98.1:39729)
0927 10:46:22.411219 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:22.411223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:22.411266 (+ 43us) server_negotiation.cc:407] Connection header received
0927 10:46:22.411410 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:22.411413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:22.411474 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:22.411601 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:46:22.411896 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.412487 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.413709 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.414314 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.414445 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:22.414635 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:22.414677 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 10:46:22.414753 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":109,"threads_started":1}
I20260927 10:46:22.415386 15097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b918621d4835470ab29b8e6fa6c32412" candidate_uuid: "50d424b2c709478ea641bdd86d39d613" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0026993090744a2973e7619c765f718" is_pre_election: true
I20260927 10:46:22.415378 14825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b918621d4835470ab29b8e6fa6c32412" candidate_uuid: "50d424b2c709478ea641bdd86d39d613" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b80217ddf067482793d1d3ab27c676d7" is_pre_election: true
I20260927 10:46:22.415690 15097 raft_consensus.cc:2454] T b918621d4835470ab29b8e6fa6c32412 P f0026993090744a2973e7619c765f718 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50d424b2c709478ea641bdd86d39d613 in term 0.
I20260927 10:46:22.415690 14825 raft_consensus.cc:2454] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50d424b2c709478ea641bdd86d39d613 in term 0.
I20260927 10:46:22.416165 14901 leader_election.cc:302] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50d424b2c709478ea641bdd86d39d613, f0026993090744a2973e7619c765f718; no voters:
I20260927 10:46:22.416436 15160 raft_consensus.cc:2781] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:46:22.416543 15160 raft_consensus.cc:482] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:46:22.416604 15160 raft_consensus.cc:3037] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:46:22.417632 15160 raft_consensus.cc:504] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } }
I20260927 10:46:22.418022 15160 leader_election.cc:288] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613 [CANDIDATE]: Term 1 election: Requested vote from peers f0026993090744a2973e7619c765f718 (127.0.98.3:42767), b80217ddf067482793d1d3ab27c676d7 (127.0.98.1:39729)
I20260927 10:46:22.418447 15097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b918621d4835470ab29b8e6fa6c32412" candidate_uuid: "50d424b2c709478ea641bdd86d39d613" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0026993090744a2973e7619c765f718"
I20260927 10:46:22.418556 14825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b918621d4835470ab29b8e6fa6c32412" candidate_uuid: "50d424b2c709478ea641bdd86d39d613" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b80217ddf067482793d1d3ab27c676d7"
I20260927 10:46:22.418601 15097 raft_consensus.cc:3037] T b918621d4835470ab29b8e6fa6c32412 P f0026993090744a2973e7619c765f718 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:46:22.418726 14825 raft_consensus.cc:3037] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:46:22.419673 15097 raft_consensus.cc:2454] T b918621d4835470ab29b8e6fa6c32412 P f0026993090744a2973e7619c765f718 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50d424b2c709478ea641bdd86d39d613 in term 1.
I20260927 10:46:22.419734 14825 raft_consensus.cc:2454] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50d424b2c709478ea641bdd86d39d613 in term 1.
I20260927 10:46:22.419977 14901 leader_election.cc:302] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50d424b2c709478ea641bdd86d39d613, f0026993090744a2973e7619c765f718; no voters:
I20260927 10:46:22.420136 15160 raft_consensus.cc:2781] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:46:22.420348 15160 raft_consensus.cc:686] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613 [term 1 LEADER]: Becoming Leader. State: Replica: 50d424b2c709478ea641bdd86d39d613, State: Running, Role: LEADER
I20260927 10:46:22.420697 15160 consensus_queue.cc:237] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } }
I20260927 10:46:22.423899 14684 catalog_manager.cc:5495] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50d424b2c709478ea641bdd86d39d613 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "50d424b2c709478ea641bdd86d39d613" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } health_report { overall_health: UNKNOWN } } }
I20260927 10:46:22.450560 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:22.447049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60572 (local address 127.0.98.2:39619)
0927 10:46:22.447199 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:22.447204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:22.447219 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:46:22.447363 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:22.447365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:22.447423 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:22.447533 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:46:22.447767 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.448272 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.449076 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:22.449295 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:22.449401 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:22.449692 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:46:22.450105 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:46:22.450238 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:22.450300 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 10:46:22.450357 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 10:46:22.456162 14825 raft_consensus.cc:1261] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [term 1 FOLLOWER]: Refusing update from remote peer 50d424b2c709478ea641bdd86d39d613: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:46:22.456163 15097 raft_consensus.cc:1261] T b918621d4835470ab29b8e6fa6c32412 P f0026993090744a2973e7619c765f718 [term 1 FOLLOWER]: Refusing update from remote peer 50d424b2c709478ea641bdd86d39d613: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:46:22.456859 15164 consensus_queue.cc:1035] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:46:22.457085 15160 consensus_queue.cc:1035] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:46:22.462484 15167 mvcc.cc:205] Tried to move back new op lower bound from 7333912504133533696 to 7333912503997689856. Current Snapshot: MvccSnapshot[applied={T|T < 7333912504133533696}]
I20260927 10:46:22.462628 15166 mvcc.cc:205] Tried to move back new op lower bound from 7333912504133533696 to 7333912503997689856. Current Snapshot: MvccSnapshot[applied={T|T < 7333912504133533696}]
I20260927 10:46:22.467479 15168 mvcc.cc:205] Tried to move back new op lower bound from 7333912504133533696 to 7333912503997689856. Current Snapshot: MvccSnapshot[applied={T|T < 7333912504133533696}]
I20260927 10:46:22.472167 14685 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:46:22.472354 14685 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:46:22.474617 14685 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb: Sending DeleteTablet for 3 replicas of tablet b918621d4835470ab29b8e6fa6c32412
I20260927 10:46:22.475379 15077 tablet_service.cc:1514] Processing DeleteTablet for tablet b918621d4835470ab29b8e6fa6c32412 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:46:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40166
I20260927 10:46:22.475375 14945 tablet_service.cc:1514] Processing DeleteTablet for tablet b918621d4835470ab29b8e6fa6c32412 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:46:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60564
I20260927 10:46:22.475517 14807 tablet_service.cc:1514] Processing DeleteTablet for tablet b918621d4835470ab29b8e6fa6c32412 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:46:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42864
I20260927 10:46:22.476096 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 14723
I20260927 10:46:22.476101 15177 tablet_replica.cc:330] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613: stopping tablet replica
I20260927 10:46:22.476449 15177 raft_consensus.cc:2229] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:46:22.476871 15177 raft_consensus.cc:2258] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:46:22.477327 15176 tablet_replica.cc:330] T b918621d4835470ab29b8e6fa6c32412 P f0026993090744a2973e7619c765f718: stopping tablet replica
I20260927 10:46:22.477610 15176 raft_consensus.cc:2229] T b918621d4835470ab29b8e6fa6c32412 P f0026993090744a2973e7619c765f718 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:46:22.477905 15176 raft_consensus.cc:2258] T b918621d4835470ab29b8e6fa6c32412 P f0026993090744a2973e7619c765f718 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:46:22.479573 15176 ts_tablet_manager.cc:1905] T b918621d4835470ab29b8e6fa6c32412 P f0026993090744a2973e7619c765f718: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:46:22.479878 15177 ts_tablet_manager.cc:1905] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:46:22.481999 15177 ts_tablet_manager.cc:1918] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:46:22.482133 15177 log.cc:1196] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/wal/wals/b918621d4835470ab29b8e6fa6c32412
I20260927 10:46:22.482462 15176 ts_tablet_manager.cc:1918] T b918621d4835470ab29b8e6fa6c32412 P f0026993090744a2973e7619c765f718: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:46:22.482523 15177 ts_tablet_manager.cc:1939] T b918621d4835470ab29b8e6fa6c32412 P 50d424b2c709478ea641bdd86d39d613: Deleting consensus metadata
I20260927 10:46:22.482597 15176 log.cc:1196] T b918621d4835470ab29b8e6fa6c32412 P f0026993090744a2973e7619c765f718: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/b918621d4835470ab29b8e6fa6c32412
I20260927 10:46:22.482935 15176 ts_tablet_manager.cc:1939] T b918621d4835470ab29b8e6fa6c32412 P f0026993090744a2973e7619c765f718: Deleting consensus metadata
I20260927 10:46:22.483803 14671 catalog_manager.cc:4841] TS 50d424b2c709478ea641bdd86d39d613 (127.0.98.2:39619): tablet b918621d4835470ab29b8e6fa6c32412 (table test-table [id=8004cac10d8545f8bb5c8cb98fd377d4]) successfully deleted
I20260927 10:46:22.484071 14672 catalog_manager.cc:4841] TS f0026993090744a2973e7619c765f718 (127.0.98.3:42767): tablet b918621d4835470ab29b8e6fa6c32412 (table test-table [id=8004cac10d8545f8bb5c8cb98fd377d4]) successfully deleted
W20260927 10:46:22.488078 14670 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39729 (error 108)
I20260927 10:46:22.488315 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 14878
I20260927 10:46:22.489827 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:22.489431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39729 (local address 127.0.0.1:42868)
0927 10:46:22.489637 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:22.489718 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260927 10:46:22.490010 14670 catalog_manager.cc:4584] TS b80217ddf067482793d1d3ab27c676d7 (127.0.98.1:39729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b918621d4835470ab29b8e6fa6c32412: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39729: connect: Connection refused (error 111)
I20260927 10:46:22.494844 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 15014
I20260927 10:46:22.500329 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 14652
I20260927 10:46:22.507117 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41583
--webserver_interface=127.0.98.62
--webserver_port=38861
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:46:22.621830 15180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:22.622102 15180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:22.622143 15180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:22.626134 15180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:46:22.626199 15180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:22.626219 15180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:22.626235 15180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:46:22.626250 15180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:46:22.631703 15180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:22.632761 15180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:22.633713 15180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:22.641889 15185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:22.641919 15186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:22.641919 15188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:22.642248 15180 server_base.cc:1034] running on GCE node
I20260927 10:46:22.642819 15180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:22.643977 15180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:22.645164 15180 hybrid_clock.cc:648] HybridClock initialized: now 1790505982645148 us; error 25 us; skew 500 ppm
Sep 27 10:46:22 dist-test-slave-dzhw krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:22.647945 15180 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:46:22.648795 15180 webserver.cc:460] Webserver started at http://127.0.98.62:38861/ using document root <none> and password file <none>
I20260927 10:46:22.649379 15180 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:22.649464 15180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:22.653160 15180 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:46:22.655228 15195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:22.656493 15180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:46:22.656646 15180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "74aa3fb48ada44db96baf018b173eefb"
format_stamp: "Formatted at 2026-09-27 10:46:20 on dist-test-slave-dzhw"
server_key: "8cf99d10a6d612f121b97bfea5bd45ed"
server_key_iv: "ed557d852b6de1e1fed9a82bd262e691"
server_key_version: "encryptionkey@0"
I20260927 10:46:22.657234 15180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:22.674013 15180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:22.677322 15180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:22.677529 15180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:22.684732 15180 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41583
I20260927 10:46:22.685019 15247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41583 every 8 connection(s)
I20260927 10:46:22.685879 15180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:46:22.689765 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 15180
I20260927 10:46:22.690198 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39729
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41583
--webserver_port=35261
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:46:22.690261 15248 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:46:22.692031 15248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb: Bootstrap starting.
I20260927 10:46:22.709477 15248 log.cc:824] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:22.720193 15248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:46:22.720587 15248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb: Bootstrap complete.
I20260927 10:46:22.724674 15248 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74aa3fb48ada44db96baf018b173eefb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41583 } }
I20260927 10:46:22.725080 15248 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74aa3fb48ada44db96baf018b173eefb, State: Initialized, Role: FOLLOWER
I20260927 10:46:22.725612 15248 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74aa3fb48ada44db96baf018b173eefb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41583 } }
I20260927 10:46:22.725754 15248 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:46:22.725840 15248 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:46:22.725952 15248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [term 2 FOLLOWER]: Advancing to term 3
I20260927 10:46:22.727921 15248 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74aa3fb48ada44db96baf018b173eefb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41583 } }
I20260927 10:46:22.728243 15248 leader_election.cc:302] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74aa3fb48ada44db96baf018b173eefb; no voters:
I20260927 10:46:22.728581 15248 leader_election.cc:288] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 10:46:22.728715 15252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [term 3 FOLLOWER]: Leader election won for term 3
I20260927 10:46:22.728946 15252 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [term 3 LEADER]: Becoming Leader. State: Replica: 74aa3fb48ada44db96baf018b173eefb, State: Running, Role: LEADER
I20260927 10:46:22.729249 15252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74aa3fb48ada44db96baf018b173eefb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41583 } }
I20260927 10:46:22.729810 15248 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:46:22.730762 15254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74aa3fb48ada44db96baf018b173eefb. Latest consensus state: current_term: 3 leader_uuid: "74aa3fb48ada44db96baf018b173eefb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74aa3fb48ada44db96baf018b173eefb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41583 } } }
I20260927 10:46:22.730796 15253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "74aa3fb48ada44db96baf018b173eefb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74aa3fb48ada44db96baf018b173eefb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41583 } } }
I20260927 10:46:22.730902 15253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [sys.catalog]: This master's current role is: LEADER
I20260927 10:46:22.730904 15254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74aa3fb48ada44db96baf018b173eefb [sys.catalog]: This master's current role is: LEADER
I20260927 10:46:22.731343 15261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:46:22.736805 15261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:46:22.737419 15261 catalog_manager.cc:1259] Loaded cluster ID: 36a62e5caf1643839c0213c6011b43b2
I20260927 10:46:22.737481 15261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 10:46:22.738446 15261 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5dd9ef19ad google::LogMessage::Fail() at ??:0
@ 0x7f5dd9ef488a google::LogMessage::SendToLog() at ??:0
@ 0x7f5dd9ef14de google::LogMessage::Flush() at ??:0
@ 0x7f5dd9ef2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5de4830fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f5de4831d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5de482e579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f5de4872640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f5ddc1433d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5ddc143c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f5ddc1453f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f5ddc134512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5ddd7786db start_thread at ??:0
@ 0x7f5dd89ef61f clone at ??:0
W20260927 10:46:22.820931 15250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:22.821236 15250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:22.821307 15250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:22.825397 15250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:22.825485 15250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:22.825572 15250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:46:22.831055 15250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35261
--tserver_master_addrs=127.0.98.62:41583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:22.832177 15250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:22.833174 15250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:22.842701 15290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:22.842768 15250 server_base.cc:1034] running on GCE node
W20260927 10:46:22.842942 15292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:22.843089 15289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:22.843509 15250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:22.844211 15250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:22.845428 15250 hybrid_clock.cc:648] HybridClock initialized: now 1790505982845399 us; error 47 us; skew 500 ppm
Sep 27 10:46:22 dist-test-slave-dzhw krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:22.848498 15250 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 10:46:22.849494 15250 webserver.cc:460] Webserver started at http://127.0.98.1:35261/ using document root <none> and password file <none>
I20260927 10:46:22.850103 15250 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:22.850168 15250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:22.855165 15250 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 10:46:22.857514 15299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:22.859072 15250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:46:22.859220 15250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "b80217ddf067482793d1d3ab27c676d7"
format_stamp: "Formatted at 2026-09-27 10:46:20 on dist-test-slave-dzhw"
server_key: "bde0fd74426ba689bfe3c8fe68596407"
server_key_iv: "e1ca6d8ea164cc9a624f710a00c48a58"
server_key_version: "encryptionkey@0"
I20260927 10:46:22.859815 15250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:22.875092 15250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:22.879149 15250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:22.879369 15250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:22.880168 15250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:46:22.881556 15306 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 10:46:22.885851 15250 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 10:46:22.885950 15250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260927 10:46:22.886041 15250 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 10:46:22.888475 15250 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 10:46:22.888546 15250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260927 10:46:22.888908 15306 tablet_bootstrap.cc:492] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7: Bootstrap starting.
I20260927 10:46:22.900182 15250 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39729
I20260927 10:46:22.900238 15413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39729 every 8 connection(s)
I20260927 10:46:22.901436 15250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:46:22.907186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 15250
I20260927 10:46:22.907709 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39619
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41583
--webserver_port=40557
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:46:22.908792 15306 log.cc:824] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:22.916049 15306 tablet_bootstrap.cc:492] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:46:22.916509 15306 tablet_bootstrap.cc:492] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7: Bootstrap complete.
I20260927 10:46:22.918098 15306 ts_tablet_manager.cc:1397] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7: Time spent bootstrapping tablet: real 0.029s user 0.017s sys 0.008s
I20260927 10:46:22.922559 15306 raft_consensus.cc:348] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } }
I20260927 10:46:22.923286 15306 raft_consensus.cc:729] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b80217ddf067482793d1d3ab27c676d7, State: Initialized, Role: FOLLOWER
I20260927 10:46:22.923904 15306 consensus_queue.cc:260] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } }
I20260927 10:46:22.925030 15306 ts_tablet_manager.cc:1428] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7: Time spent starting tablet: real 0.007s user 0.010s sys 0.000s
W20260927 10:46:22.975459 15416 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:46:22.903824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.1:60305)
0927 10:46:22.904409 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:22.904467 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:22.905656 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:22.974685 (+ 69029us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":158,"threads_started":1}
W20260927 10:46:22.976063 15414 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41583 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41583: Client connection negotiation failed: client connection to 127.0.98.62:41583: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 10:46:22.976809 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:22.976550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.1:52175)
0927 10:46:22.976670 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:22.976751 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 10:46:22.977473 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:22.977278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.1:55905)
0927 10:46:22.977375 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:22.977419 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 10:46:22.977694 15414 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:46:23.028184 15417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:23.028482 15417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:23.028554 15417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:23.032598 15417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:23.032703 15417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:23.032795 15417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:46:23.038177 15417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39619
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40557
--tserver_master_addrs=127.0.98.62:41583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:23.039364 15417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:23.040354 15417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:23.050057 15425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:23.050040 15424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:23.050174 15417 server_base.cc:1034] running on GCE node
W20260927 10:46:23.050375 15427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:23.050813 15417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:23.051436 15417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:23.052621 15417 hybrid_clock.cc:648] HybridClock initialized: now 1790505983052610 us; error 46 us; skew 500 ppm
Sep 27 10:46:23 dist-test-slave-dzhw krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:23.055300 15417 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 10:46:23.056149 15417 webserver.cc:460] Webserver started at http://127.0.98.2:40557/ using document root <none> and password file <none>
I20260927 10:46:23.056706 15417 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:23.056795 15417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:23.060567 15417 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 10:46:23.062602 15434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:23.063836 15417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:46:23.064006 15417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "50d424b2c709478ea641bdd86d39d613"
format_stamp: "Formatted at 2026-09-27 10:46:20 on dist-test-slave-dzhw"
server_key: "ad962da6a91e6657f5530acb01f81d59"
server_key_iv: "6953b5d0dcfe896534ec48074e0334fe"
server_key_version: "encryptionkey@0"
I20260927 10:46:23.064635 15417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:23.089493 15417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:23.092523 15417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:23.092725 15417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:23.093341 15417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:46:23.094180 15417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:46:23.094228 15417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:23.094269 15417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:46:23.094323 15417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:23.103765 15417 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39619
I20260927 10:46:23.103788 15547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39619 every 8 connection(s)
I20260927 10:46:23.104773 15417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:46:23.108495 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:23.107006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.2:48341)
0927 10:46:23.107680 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:23.107788 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":115,"threads_started":1}
W20260927 10:46:23.108964 15548 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41583 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41583: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
I20260927 10:46:23.109427 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 15417
I20260927 10:46:23.109644 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:23.109370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.2:48103)
0927 10:46:23.109498 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:23.109583 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 10:46:23.109934 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42767
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41583
--webserver_port=38365
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:46:23.110316 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:23.110110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.2:36169)
0927 10:46:23.110219 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:23.110263 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 10:46:23.110513 15548 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:46:23.219172 15551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:23.219475 15551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:23.219553 15551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:23.223575 15551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:23.223659 15551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:23.223747 15551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:46:23.228968 15551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38365
--tserver_master_addrs=127.0.98.62:41583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:23.230072 15551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:23.231096 15551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:23.240355 15556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:23.240391 15557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:23.240427 15559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:23.240804 15551 server_base.cc:1034] running on GCE node
I20260927 10:46:23.241204 15551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:23.241837 15551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:23.243077 15551 hybrid_clock.cc:648] HybridClock initialized: now 1790505983243051 us; error 39 us; skew 500 ppm
Sep 27 10:46:23 dist-test-slave-dzhw krb5kdc[14105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:23.246008 15551 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 10:46:23.247013 15551 webserver.cc:460] Webserver started at http://127.0.98.3:38365/ using document root <none> and password file <none>
I20260927 10:46:23.247583 15551 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:23.247649 15551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:23.251724 15551 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:46:23.254493 15566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:23.255801 15551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:46:23.255947 15551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "f0026993090744a2973e7619c765f718"
format_stamp: "Formatted at 2026-09-27 10:46:20 on dist-test-slave-dzhw"
server_key: "6c1b7c4dd76015682138143ecd389d2c"
server_key_iv: "57813eb4f82ed73ee29d29d7345620b4"
server_key_version: "encryptionkey@0"
I20260927 10:46:23.256529 15551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:23.278716 15551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:23.281653 15551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:23.281855 15551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:23.282557 15551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:46:23.283363 15551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:46:23.283438 15551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:23.283505 15551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:46:23.283538 15551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:23.292493 15551 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42767
I20260927 10:46:23.292521 15679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42767 every 8 connection(s)
I20260927 10:46:23.293501 15551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:46:23.296304 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 15551
I20260927 10:46:23.297364 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:23.295664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.3:37091)
0927 10:46:23.296455 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:23.296565 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":134,"threads_started":1}
W20260927 10:46:23.297811 15680 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41583 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41583: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
I20260927 10:46:23.298400 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:23.298178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.3:41143)
0927 10:46:23.298301 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:23.298348 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 10:46:23.299149 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:23.298900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.3:48729)
0927 10:46:23.299053 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:23.299100 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 10:46:23.299394 15680 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 10:46:23.978736 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:23.978249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.1:60949)
0927 10:46:23.978563 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:23.978619 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":74,"threads_started":1}
I20260927 10:46:24.111411 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:24.111039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.2:43517)
0927 10:46:24.111265 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:24.111318 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":84,"threads_started":1}
I20260927 10:46:24.300503 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:24.300032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.3:49107)
0927 10:46:24.300320 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:24.300386 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":75,"threads_started":1}
I20260927 10:46:24.420274 15687 raft_consensus.cc:482] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:46:24.420470 15687 raft_consensus.cc:504] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } }
I20260927 10:46:24.421638 15687 leader_election.cc:288] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0026993090744a2973e7619c765f718 (127.0.98.3:42767), 50d424b2c709478ea641bdd86d39d613 (127.0.98.2:39619)
Sep 27 10:46:24 dist-test-slave-dzhw krb5kdc[14105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 27 10:46:24 dist-test-slave-dzhw krb5kdc[14105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 10:46:24.435840 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:24.422010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42767 (local address 127.0.98.1:40251)
0927 10:46:24.422545 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:24.422561 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:24.422660 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:24.424452 (+ 1792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:24.424457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:24.424778 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:24.425142 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:24.425157 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:24.426451 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:24.426455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:24.427389 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:24.427399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:24.428110 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:24.428113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:24.428272 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:24.429043 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:24.429059 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:24.430880 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:24.433283 (+ 2403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:24.433291 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:24.433308 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:24.433680 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:24.434072 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:24.434078 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:24.434083 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:24.434240 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:24.435067 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:24.435070 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:24.435228 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 10:46:24.435410 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 10:46:24.435683 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":221,"spinlock_wait_cycles":11008,"thread_start_us":87,"threads_started":1}
I20260927 10:46:24.436007 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:24.422129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47211 (local address 127.0.98.2:39619)
0927 10:46:24.422578 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:24.422607 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:24.422656 (+ 49us) server_negotiation.cc:407] Connection header received
0927 10:46:24.423407 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:24.423413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:24.423725 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:24.424054 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:24.425329 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:24.426373 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:24.427529 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:24.428023 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:24.428218 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:24.431507 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:24.431511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:24.431522 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:24.431553 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:24.433374 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:24.433879 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:24.433883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:24.433887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:24.433990 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:24.434337 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:24.434340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:24.434342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:24.434812 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:24.434957 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:24.435546 (+ 589us) server_negotiation.cc:299] Negotiation successful
0927 10:46:24.435828 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":132,"threads_started":1}
I20260927 10:46:24.436254 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:24.422000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39619 (local address 127.0.98.1:47211)
0927 10:46:24.422120 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:24.422138 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:24.422253 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:24.424114 (+ 1861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:24.424121 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:24.424679 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:24.425143 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:24.425158 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:24.426563 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:24.426567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:24.427392 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:24.427402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:24.428161 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:24.428165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:24.428277 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:24.429114 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:24.429128 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:24.431307 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:24.433503 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:24.433506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:24.433507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:24.433775 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:24.434139 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:24.434142 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:24.434143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:24.434240 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:24.434945 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:24.434951 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:24.435230 (+ 279us) client_negotiation.cc:769] Sending connection context
0927 10:46:24.435438 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 10:46:24.435782 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":165,"spinlock_wait_cycles":13184}
I20260927 10:46:24.436800 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:24.422287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40251 (local address 127.0.98.3:42767)
0927 10:46:24.422610 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:24.422629 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:24.422661 (+ 32us) server_negotiation.cc:407] Connection header received
0927 10:46:24.423379 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:24.423385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:24.423725 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:24.424050 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:24.425309 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:24.426283 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:24.427514 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:24.427978 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:24.428158 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:24.431170 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:24.431176 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:24.431192 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:24.431221 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:24.433146 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:24.433804 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:24.433812 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:24.433820 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:24.433958 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:24.434343 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:24.434347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:24.434348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:24.434883 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:24.435017 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:24.436348 (+ 1331us) server_negotiation.cc:299] Negotiation successful
0927 10:46:24.436636 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":74,"threads_started":1}
W20260927 10:46:24.438795 15303 leader_election.cc:341] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0026993090744a2973e7619c765f718 (127.0.98.3:42767): Not found: Tablet not found: b918621d4835470ab29b8e6fa6c32412
W20260927 10:46:24.438961 15302 leader_election.cc:341] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 50d424b2c709478ea641bdd86d39d613 (127.0.98.2:39619): Not found: Tablet not found: b918621d4835470ab29b8e6fa6c32412
I20260927 10:46:24.439062 15302 leader_election.cc:302] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b80217ddf067482793d1d3ab27c676d7; no voters: 50d424b2c709478ea641bdd86d39d613, f0026993090744a2973e7619c765f718
I20260927 10:46:24.439355 15687 raft_consensus.cc:2726] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:46:24.979956 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:24.979533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.1:44373)
0927 10:46:24.979800 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:24.979859 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":75,"threads_started":1}
I20260927 10:46:25.112516 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:25.112117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.2:54117)
0927 10:46:25.112371 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:25.112428 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":77,"threads_started":1}
I20260927 10:46:25.301800 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:25.301344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.3:38059)
0927 10:46:25.301623 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:25.301682 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":102,"threads_started":1}
I20260927 10:46:25.981220 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:25.980767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.1:60457)
0927 10:46:25.981013 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:25.981077 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":90,"threads_started":1}
I20260927 10:46:26.113626 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:26.113240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.2:53971)
0927 10:46:26.113468 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:26.113518 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":72,"threads_started":1}
I20260927 10:46:26.303196 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:26.302648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.3:49929)
0927 10:46:26.302992 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:26.303075 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":72,"threads_started":1}
I20260927 10:46:26.639570 15697 raft_consensus.cc:482] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:46:26.639707 15697 raft_consensus.cc:504] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } }
I20260927 10:46:26.640142 15697 leader_election.cc:288] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0026993090744a2973e7619c765f718 (127.0.98.3:42767), 50d424b2c709478ea641bdd86d39d613 (127.0.98.2:39619)
W20260927 10:46:26.640969 15302 leader_election.cc:341] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 50d424b2c709478ea641bdd86d39d613 (127.0.98.2:39619): Not found: Tablet not found: b918621d4835470ab29b8e6fa6c32412
W20260927 10:46:26.641073 15303 leader_election.cc:341] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0026993090744a2973e7619c765f718 (127.0.98.3:42767): Not found: Tablet not found: b918621d4835470ab29b8e6fa6c32412
I20260927 10:46:26.641196 15303 leader_election.cc:302] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b80217ddf067482793d1d3ab27c676d7; no voters: 50d424b2c709478ea641bdd86d39d613, f0026993090744a2973e7619c765f718
I20260927 10:46:26.641340 15697 raft_consensus.cc:2726] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:46:26.982473 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:26.982003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.1:34099)
0927 10:46:26.982263 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:26.982318 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":95,"threads_started":1}
I20260927 10:46:27.114940 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:27.114513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.2:41889)
0927 10:46:27.114781 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:27.114842 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":79,"threads_started":1}
I20260927 10:46:27.304486 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:27.304036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.3:60067)
0927 10:46:27.304325 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:27.304385 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":82,"threads_started":1}
I20260927 10:46:27.983867 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:27.983404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.1:60177)
0927 10:46:27.983711 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:27.983770 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":75,"threads_started":1}
I20260927 10:46:28.116232 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:28.115784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.2:54219)
0927 10:46:28.116075 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:28.116137 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":79,"threads_started":1}
I20260927 10:46:28.305810 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:28.305380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.3:35525)
0927 10:46:28.305667 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:28.305720 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":94,"threads_started":1}
I20260927 10:46:28.985117 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:28.984736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.1:35581)
0927 10:46:28.984971 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:28.985020 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
I20260927 10:46:29.117373 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:29.116956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.2:36559)
0927 10:46:29.117220 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:29.117275 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":85,"threads_started":1}
I20260927 10:46:29.307104 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:29.306714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.3:52477)
0927 10:46:29.306952 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:29.307014 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
I20260927 10:46:29.573416 15707 raft_consensus.cc:482] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:46:29.573549 15707 raft_consensus.cc:504] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } }
I20260927 10:46:29.574121 15707 leader_election.cc:288] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0026993090744a2973e7619c765f718 (127.0.98.3:42767), 50d424b2c709478ea641bdd86d39d613 (127.0.98.2:39619)
W20260927 10:46:29.574986 15302 leader_election.cc:341] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 50d424b2c709478ea641bdd86d39d613 (127.0.98.2:39619): Not found: Tablet not found: b918621d4835470ab29b8e6fa6c32412
W20260927 10:46:29.575083 15303 leader_election.cc:341] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0026993090744a2973e7619c765f718 (127.0.98.3:42767): Not found: Tablet not found: b918621d4835470ab29b8e6fa6c32412
I20260927 10:46:29.575201 15303 leader_election.cc:302] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b80217ddf067482793d1d3ab27c676d7; no voters: 50d424b2c709478ea641bdd86d39d613, f0026993090744a2973e7619c765f718
I20260927 10:46:29.575364 15707 raft_consensus.cc:2726] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:46:29.986361 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:29.985903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.1:52773)
0927 10:46:29.986197 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:29.986255 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":77,"threads_started":1}
I20260927 10:46:30.118747 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:30.118209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.2:42713)
0927 10:46:30.118589 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:30.118652 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":78,"threads_started":1}
I20260927 10:46:30.308540 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:30.307951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.3:43147)
0927 10:46:30.308311 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:30.308402 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":110,"threads_started":1}
I20260927 10:46:30.987577 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:30.987161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.1:48377)
0927 10:46:30.987413 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:30.987469 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":78,"threads_started":1}
I20260927 10:46:31.119944 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:31.119532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.2:54663)
0927 10:46:31.119790 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:31.119850 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
I20260927 10:46:31.309967 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:31.309471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.3:58073)
0927 10:46:31.309764 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:31.309837 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":80,"threads_started":1}
I20260927 10:46:31.988835 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:31.988378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.1:39929)
0927 10:46:31.988671 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:31.988734 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":86,"threads_started":1}
I20260927 10:46:32.121407 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:32.120782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.2:56277)
0927 10:46:32.121145 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:32.121216 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":120,"threads_started":1}
I20260927 10:46:32.311370 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:32.310947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.3:38095)
0927 10:46:32.311217 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:32.311275 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":77,"threads_started":1}
I20260927 10:46:32.990121 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:32.989740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.1:44731)
0927 10:46:32.989973 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:32.990024 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":76,"threads_started":1}
I20260927 10:46:33.122666 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:33.122151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.2:44253)
0927 10:46:33.122499 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:33.122558 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":87,"threads_started":1}
I20260927 10:46:33.312616 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:33.312208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.3:53011)
0927 10:46:33.312464 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:33.312523 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":78,"threads_started":1}
I20260927 10:46:33.991392 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:33.990975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.1:52145)
0927 10:46:33.991235 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:33.991295 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
I20260927 10:46:34.042551 15721 raft_consensus.cc:482] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:46:34.042704 15721 raft_consensus.cc:504] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0026993090744a2973e7619c765f718" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42767 } } peers { permanent_uuid: "50d424b2c709478ea641bdd86d39d613" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39619 } } peers { permanent_uuid: "b80217ddf067482793d1d3ab27c676d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39729 } }
I20260927 10:46:34.043201 15721 leader_election.cc:288] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0026993090744a2973e7619c765f718 (127.0.98.3:42767), 50d424b2c709478ea641bdd86d39d613 (127.0.98.2:39619)
W20260927 10:46:34.043948 15303 leader_election.cc:341] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0026993090744a2973e7619c765f718 (127.0.98.3:42767): Not found: Tablet not found: b918621d4835470ab29b8e6fa6c32412
W20260927 10:46:34.044077 15302 leader_election.cc:341] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 50d424b2c709478ea641bdd86d39d613 (127.0.98.2:39619): Not found: Tablet not found: b918621d4835470ab29b8e6fa6c32412
I20260927 10:46:34.044183 15302 leader_election.cc:302] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b80217ddf067482793d1d3ab27c676d7; no voters: 50d424b2c709478ea641bdd86d39d613, f0026993090744a2973e7619c765f718
I20260927 10:46:34.044337 15721 raft_consensus.cc:2726] T b918621d4835470ab29b8e6fa6c32412 P b80217ddf067482793d1d3ab27c676d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:46:34.123911 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:34.123475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.2:40833)
0927 10:46:34.123753 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:34.123813 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":92,"threads_started":1}
I20260927 10:46:34.313968 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:34.313513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.3:54225)
0927 10:46:34.313797 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:34.313861 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":79,"threads_started":1}
I20260927 10:46:34.992849 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:34.992426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.1:39035)
0927 10:46:34.992685 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:34.992746 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":81,"threads_started":1}
I20260927 10:46:35.125137 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:35.124708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.2:38559)
0927 10:46:35.124968 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:35.125038 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
I20260927 10:46:35.315162 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:35.314760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.3:57285)
0927 10:46:35.315019 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:35.315070 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":88,"threads_started":1}
I20260927 10:46:35.994076 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:35.993699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.1:45711)
0927 10:46:35.993929 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:35.993979 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
I20260927 10:46:36.126353 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:36.125949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.2:57289)
0927 10:46:36.126199 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:36.126257 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":76,"threads_started":1}
I20260927 10:46:36.316381 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:36.315952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.3:50271)
0927 10:46:36.316221 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:36.316286 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":94,"threads_started":1}
I20260927 10:46:36.995410 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:36.994975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.1:54131)
0927 10:46:36.995236 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:36.995297 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":78,"threads_started":1}
I20260927 10:46:37.127671 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:37.127215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.2:48225)
0927 10:46:37.127507 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:37.127571 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":83,"threads_started":1}
I20260927 10:46:37.317728 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:37.317260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.3:53573)
0927 10:46:37.317544 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:37.317631 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":95,"threads_started":1}
I20260927 10:46:37.996884 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:37.996475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.1:50611)
0927 10:46:37.996728 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:37.996787 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":72,"threads_started":1}
I20260927 10:46:38.129144 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:38.128686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41583 (local address 127.0.98.2:42127)
0927 10:46:38.128988 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:38.129048 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":79,"threads_started":1}
I20260927 10:46:38.305584 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 15250
I20260927 10:46:38.311511 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 15417
I20260927 10:46:38.317034 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 15551
2026-09-27T10:46:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19407 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:46:38 dist-test-slave-dzhw krb5kdc[15737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:46:38 dist-test-slave-dzhw krb5kdc[15737](info): set up 2 sockets
Sep 27 10:46:38 dist-test-slave-dzhw krb5kdc[15737](info): commencing operation
krb5kdc: starting...
W20260927 10:46:39.362730 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:46:39 dist-test-slave-dzhw krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:46:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:46:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:46:39.486333 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37323
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:46:39.596081 15750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:39.596360 15750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:39.596410 15750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:39.600327 15750 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 10:46:39.600383 15750 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 10:46:39.600400 15750 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 10:46:39.600414 15750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:46:39.600432 15750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:39.600449 15750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:39.600464 15750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:46:39.600479 15750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:46:39.605743 15750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:39.606783 15750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:39.607743 15750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:39.616371 15758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:39.616379 15756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:39.616439 15755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:39.616658 15750 server_base.cc:1034] running on GCE node
I20260927 10:46:39.617236 15750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:39.618340 15750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:39.619524 15750 hybrid_clock.cc:648] HybridClock initialized: now 1790505999619507 us; error 32 us; skew 500 ppm
Sep 27 10:46:39 dist-test-slave-dzhw krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:39.622403 15750 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:46:39.623299 15750 webserver.cc:460] Webserver started at http://127.0.98.62:42739/ using document root <none> and password file <none>
I20260927 10:46:39.623816 15750 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:39.623881 15750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:39.624042 15750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:46:39.626015 15750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "52f42c39e2e74738bed67e4222acad73"
format_stamp: "Formatted at 2026-09-27 10:46:39 on dist-test-slave-dzhw"
server_key: "2cd0c87e967318a860efbe877a7ce38b"
server_key_iv: "b77921142fad963c2fd504e7a49e8a9c"
server_key_version: "encryptionkey@0"
I20260927 10:46:39.626576 15750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "52f42c39e2e74738bed67e4222acad73"
format_stamp: "Formatted at 2026-09-27 10:46:39 on dist-test-slave-dzhw"
server_key: "2cd0c87e967318a860efbe877a7ce38b"
server_key_iv: "b77921142fad963c2fd504e7a49e8a9c"
server_key_version: "encryptionkey@0"
I20260927 10:46:39.630218 15750 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:46:39.632664 15765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:39.633911 15750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:46:39.634066 15750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "52f42c39e2e74738bed67e4222acad73"
format_stamp: "Formatted at 2026-09-27 10:46:39 on dist-test-slave-dzhw"
server_key: "2cd0c87e967318a860efbe877a7ce38b"
server_key_iv: "b77921142fad963c2fd504e7a49e8a9c"
server_key_version: "encryptionkey@0"
I20260927 10:46:39.634184 15750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:39.653054 15750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:39.656247 15750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:39.656471 15750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:39.663748 15750 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37323
I20260927 10:46:39.664036 15817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37323 every 8 connection(s)
I20260927 10:46:39.664840 15750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:46:39.667574 15818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:46:39.672791 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 15750
I20260927 10:46:39.672914 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:46:39.673177 15818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73: Bootstrap starting.
I20260927 10:46:39.673203 392 external_mini_cluster.cc:1428] Setting key 06fae254bc5932824ac594ad5056c9a1
I20260927 10:46:39.675750 15818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73: Neither blocks nor log segments found. Creating new log.
I20260927 10:46:39.676489 15818 log.cc:824] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:39.678329 15818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73: No bootstrap required, opened a new log
Sep 27 10:46:39 dist-test-slave-dzhw krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:46:39.682075 15818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f42c39e2e74738bed67e4222acad73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37323 } }
I20260927 10:46:39.682330 15818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:46:39.682406 15818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f42c39e2e74738bed67e4222acad73, State: Initialized, Role: FOLLOWER
I20260927 10:46:39.682976 15818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f42c39e2e74738bed67e4222acad73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37323 } }
I20260927 10:46:39.683154 15818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:46:39.683243 15818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:46:39.683351 15818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:46:39.684283 15818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f42c39e2e74738bed67e4222acad73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37323 } }
I20260927 10:46:39.684628 15818 leader_election.cc:302] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52f42c39e2e74738bed67e4222acad73; no voters:
I20260927 10:46:39.685001 15818 leader_election.cc:288] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:46:39.685072 15823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:46:39.685286 15823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [term 1 LEADER]: Becoming Leader. State: Replica: 52f42c39e2e74738bed67e4222acad73, State: Running, Role: LEADER
I20260927 10:46:39.685655 15823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f42c39e2e74738bed67e4222acad73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37323 } }
I20260927 10:46:39.686396 15818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:46:39.687047 15823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52f42c39e2e74738bed67e4222acad73. Latest consensus state: current_term: 1 leader_uuid: "52f42c39e2e74738bed67e4222acad73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f42c39e2e74738bed67e4222acad73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37323 } } }
I20260927 10:46:39.687254 15823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [sys.catalog]: This master's current role is: LEADER
I20260927 10:46:39.687463 15824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52f42c39e2e74738bed67e4222acad73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f42c39e2e74738bed67e4222acad73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37323 } } }
I20260927 10:46:39.687629 15824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [sys.catalog]: This master's current role is: LEADER
I20260927 10:46:39.687989 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:39.674706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44848 (local address 127.0.98.62:37323)
0927 10:46:39.675129 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:39.675144 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:39.675178 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:46:39.675823 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:39.675840 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:39.676221 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:39.676553 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:39.677316 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:39.678278 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:39.679159 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:39.679612 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:39.679781 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:39.682217 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:39.682222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:39.682236 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:39.682263 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:39.684168 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:39.684649 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:39.684655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:39.684662 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:39.684836 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:39.685176 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:39.685179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:39.685181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:39.685696 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:39.685965 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:39.686305 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 10:46:39.686899 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":127,"threads_started":1}
W20260927 10:46:39.691473 15838 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:46:39.691557 15838 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:46:39.691614 15829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:46:39.692478 15829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:46:39.698239 15829 catalog_manager.cc:1347] Generated new cluster ID: 8bd3389dea394e959f1a07f02b555330
I20260927 10:46:39.698334 15829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:46:39.713213 15829 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:46:39.714507 15829 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:46:39.723393 15829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73: Generated new TSK 0
I20260927 10:46:39.724128 15829 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 10:46:39.761490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37323
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:46:39.903757 15844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:39.904057 15844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:39.904104 15844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:39.908278 15844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:39.908386 15844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:39.908493 15844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:46:39.913852 15844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:39.915071 15844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:39.916126 15844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:46:39.925930 15844 server_base.cc:1034] running on GCE node
W20260927 10:46:39.925892 15852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:39.925872 15849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:39.925882 15850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:39.926651 15844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:39.927318 15844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:39.928544 15844 hybrid_clock.cc:648] HybridClock initialized: now 1790505999928512 us; error 51 us; skew 500 ppm
Sep 27 10:46:39 dist-test-slave-dzhw krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:39.931556 15844 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 10:46:39.932570 15844 webserver.cc:460] Webserver started at http://127.0.98.1:36605/ using document root <none> and password file <none>
I20260927 10:46:39.933166 15844 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:39.933259 15844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:39.933502 15844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:46:39.935693 15844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "a5663ccff76448bab7a0118f4d2d315f"
format_stamp: "Formatted at 2026-09-27 10:46:39 on dist-test-slave-dzhw"
server_key: "c23aa5f65c6d5336923cfd3a4519e335"
server_key_iv: "25130d513879fb8f7a78f07e1dc1b4f4"
server_key_version: "encryptionkey@0"
I20260927 10:46:39.936265 15844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a5663ccff76448bab7a0118f4d2d315f"
format_stamp: "Formatted at 2026-09-27 10:46:39 on dist-test-slave-dzhw"
server_key: "c23aa5f65c6d5336923cfd3a4519e335"
server_key_iv: "25130d513879fb8f7a78f07e1dc1b4f4"
server_key_version: "encryptionkey@0"
I20260927 10:46:39.940517 15844 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:46:39.943015 15859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:39.944239 15844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:46:39.944401 15844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "a5663ccff76448bab7a0118f4d2d315f"
format_stamp: "Formatted at 2026-09-27 10:46:39 on dist-test-slave-dzhw"
server_key: "c23aa5f65c6d5336923cfd3a4519e335"
server_key_iv: "25130d513879fb8f7a78f07e1dc1b4f4"
server_key_version: "encryptionkey@0"
I20260927 10:46:39.944526 15844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:39.964237 15844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:39.967742 15844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:39.967960 15844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:39.968626 15844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:46:39.969511 15844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:46:39.969581 15844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:39.969653 15844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:46:39.969695 15844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:39.979121 15844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42477
I20260927 10:46:39.979138 15972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42477 every 8 connection(s)
I20260927 10:46:39.980041 15844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:46:39.989078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 15844
I20260927 10:46:39.989183 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:46:39.989472 392 external_mini_cluster.cc:1428] Setting key e8108fdc7647791cb816d7106f33c91f
Sep 27 10:46:39 dist-test-slave-dzhw krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:46:39.994242 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:39.981851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43889 (local address 127.0.98.62:37323)
0927 10:46:39.981998 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:39.982003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:39.983052 (+ 1049us) server_negotiation.cc:407] Connection header received
0927 10:46:39.983930 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:39.983932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:39.983978 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:39.984056 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:39.985214 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:39.985773 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:39.987027 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:39.987320 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:39.987413 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:39.990080 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:39.990084 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:39.990086 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:39.990114 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:39.991769 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:39.992306 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:39.992313 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:39.992318 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:39.992412 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:39.992830 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:39.992837 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:39.992840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:39.993046 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:39.993122 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:39.993844 (+ 722us) server_negotiation.cc:299] Negotiation successful
0927 10:46:39.993972 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:46:39.995162 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:39.982216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.1:43889)
0927 10:46:39.982919 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:39.982964 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:39.983732 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:39.984217 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:39.984224 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:39.984599 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:39.985067 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:39.985081 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:39.985893 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:39.985895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:39.986873 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:39.986882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:39.987505 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:39.987513 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:39.987680 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:39.988360 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:39.988376 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:39.989903 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:39.991910 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:39.991915 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:39.991928 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:39.992203 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:39.992506 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:39.992508 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:39.992509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:39.992712 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:39.993154 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:39.993162 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:39.993544 (+ 382us) client_negotiation.cc:769] Sending connection context
0927 10:46:39.993788 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 10:46:39.994178 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":179,"threads_started":1}
I20260927 10:46:39.996447 15973 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37323
I20260927 10:46:39.996740 15973 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:39.997398 15973 heartbeater.cc:511] Master 127.0.98.62:37323 requested a full tablet report, sending...
I20260927 10:46:39.999035 15778 ts_manager.cc:194] Registered new tserver with Master: a5663ccff76448bab7a0118f4d2d315f (127.0.98.1:42477)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 10:46:40.000255 15778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43889
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 10:46:40.019620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37323
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:46:40.131830 15978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:40.132144 15978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:40.132215 15978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:40.136369 15978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:40.136427 15978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:40.136494 15978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:46:40.141956 15978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:40.143030 15978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:40.143989 15978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:40.153512 15983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:40.153527 15984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:40.153618 15978 server_base.cc:1034] running on GCE node
W20260927 10:46:40.153906 15986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:40.154291 15978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:40.155031 15978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:40.156214 15978 hybrid_clock.cc:648] HybridClock initialized: now 1790506000156201 us; error 39 us; skew 500 ppm
Sep 27 10:46:40 dist-test-slave-dzhw krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:40.159015 15978 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 10:46:40.159929 15978 webserver.cc:460] Webserver started at http://127.0.98.2:33287/ using document root <none> and password file <none>
I20260927 10:46:40.160439 15978 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:40.160506 15978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:40.160669 15978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:46:40.162717 15978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "e22505803753467296b25b5d6ea2f927"
format_stamp: "Formatted at 2026-09-27 10:46:40 on dist-test-slave-dzhw"
server_key: "fe145c6006e916b10346244a0eddab0b"
server_key_iv: "cb08fb4a940d25a44887b8fc67054933"
server_key_version: "encryptionkey@0"
I20260927 10:46:40.163214 15978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e22505803753467296b25b5d6ea2f927"
format_stamp: "Formatted at 2026-09-27 10:46:40 on dist-test-slave-dzhw"
server_key: "fe145c6006e916b10346244a0eddab0b"
server_key_iv: "cb08fb4a940d25a44887b8fc67054933"
server_key_version: "encryptionkey@0"
I20260927 10:46:40.166931 15978 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 10:46:40.169251 15993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:40.170655 15978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:46:40.170758 15978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "e22505803753467296b25b5d6ea2f927"
format_stamp: "Formatted at 2026-09-27 10:46:40 on dist-test-slave-dzhw"
server_key: "fe145c6006e916b10346244a0eddab0b"
server_key_iv: "cb08fb4a940d25a44887b8fc67054933"
server_key_version: "encryptionkey@0"
I20260927 10:46:40.170847 15978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:40.186058 15978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:40.190279 15978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:40.190539 15978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:40.191135 15978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:46:40.191977 15978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:46:40.192049 15978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:40.192116 15978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:46:40.192145 15978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:40.200712 15978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40283
I20260927 10:46:40.200768 16106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40283 every 8 connection(s)
I20260927 10:46:40.201644 15978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:46:40.206384 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 15978
I20260927 10:46:40.206605 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:46:40.206915 392 external_mini_cluster.cc:1428] Setting key d43e764a2cc33c9b296c0e6024f78121
Sep 27 10:46:40 dist-test-slave-dzhw krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:46:40.217724 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:40.203479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58453 (local address 127.0.98.62:37323)
0927 10:46:40.203631 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:40.203636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:40.204988 (+ 1352us) server_negotiation.cc:407] Connection header received
0927 10:46:40.205934 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:40.205937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:40.205993 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:40.206079 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:40.207716 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.208449 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.209712 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.210139 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.210245 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:40.213159 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:40.213163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:40.213166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:40.213198 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:40.215146 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:40.215700 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:40.215705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:40.215709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:40.215814 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:40.216170 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:40.216174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:40.216176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:40.216470 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:40.216587 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:40.217295 (+ 708us) server_negotiation.cc:299] Negotiation successful
0927 10:46:40.217487 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 10:46:40.218626 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:40.203896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.2:58453)
0927 10:46:40.204816 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:40.204870 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:40.205694 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:40.206259 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:40.206266 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:40.206847 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:40.207563 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:40.207582 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.208580 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.208583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:40.209575 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:40.209585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.210326 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.210333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:40.210609 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:40.211344 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:40.211373 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:40.212971 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:40.215279 (+ 2308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:40.215284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:40.215298 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:40.215584 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:40.215926 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:40.215928 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:40.215929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:40.216066 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:40.216615 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:40.216621 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:40.216951 (+ 330us) client_negotiation.cc:769] Sending connection context
0927 10:46:40.217210 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 10:46:40.217577 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":538,"threads_started":1}
I20260927 10:46:40.219878 16107 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37323
I20260927 10:46:40.220166 16107 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:40.220732 16107 heartbeater.cc:511] Master 127.0.98.62:37323 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 10:46:40.221827 15778 ts_manager.cc:194] Registered new tserver with Master: e22505803753467296b25b5d6ea2f927 (127.0.98.2:40283)
I20260927 10:46:40.222402 15778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 10:46:40.237946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37323
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:46:40.350719 16112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:40.351007 16112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:40.351089 16112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:40.355247 16112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:40.355334 16112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:40.355433 16112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:46:40.360778 16112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:46:40.361850 16112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:40.362982 16112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:40.372953 16120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:40.373018 16112 server_base.cc:1034] running on GCE node
W20260927 10:46:40.372962 16118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:40.372962 16117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:40.373778 16112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:40.374516 16112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:40.375699 16112 hybrid_clock.cc:648] HybridClock initialized: now 1790506000375685 us; error 26 us; skew 500 ppm
Sep 27 10:46:40 dist-test-slave-dzhw krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:40.378400 16112 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 10:46:40.379317 16112 webserver.cc:460] Webserver started at http://127.0.98.3:37773/ using document root <none> and password file <none>
I20260927 10:46:40.379904 16112 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:40.379995 16112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:40.380208 16112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:46:40.382285 16112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "fc6fe33c2104438eb903f1ec0d89f221"
format_stamp: "Formatted at 2026-09-27 10:46:40 on dist-test-slave-dzhw"
server_key: "c291b54b396fe0e92a0e1fab372a57a7"
server_key_iv: "dc55ad3b0ed46668b4300a51129b235b"
server_key_version: "encryptionkey@0"
I20260927 10:46:40.382918 16112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fc6fe33c2104438eb903f1ec0d89f221"
format_stamp: "Formatted at 2026-09-27 10:46:40 on dist-test-slave-dzhw"
server_key: "c291b54b396fe0e92a0e1fab372a57a7"
server_key_iv: "dc55ad3b0ed46668b4300a51129b235b"
server_key_version: "encryptionkey@0"
I20260927 10:46:40.386673 16112 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:46:40.389334 16127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:40.390605 16112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:46:40.390748 16112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "fc6fe33c2104438eb903f1ec0d89f221"
format_stamp: "Formatted at 2026-09-27 10:46:40 on dist-test-slave-dzhw"
server_key: "c291b54b396fe0e92a0e1fab372a57a7"
server_key_iv: "dc55ad3b0ed46668b4300a51129b235b"
server_key_version: "encryptionkey@0"
I20260927 10:46:40.390868 16112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:40.401232 16112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:40.404450 16112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:40.404681 16112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:40.405275 16112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:46:40.406104 16112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:46:40.406175 16112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:40.406250 16112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:46:40.406292 16112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:40.415464 16112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36377
I20260927 10:46:40.415485 16240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36377 every 8 connection(s)
I20260927 10:46:40.416353 16112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:46:40.424806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 16112
I20260927 10:46:40.424971 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:46:40.425295 392 external_mini_cluster.cc:1428] Setting key e8bb9f611345cac3002435811d007d8d
Sep 27 10:46:40 dist-test-slave-dzhw krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:46:40.429486 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:40.418147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56553 (local address 127.0.98.62:37323)
0927 10:46:40.418319 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:40.418324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:40.419390 (+ 1066us) server_negotiation.cc:407] Connection header received
0927 10:46:40.420213 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:40.420216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:40.420260 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:40.420337 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:40.421485 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.422009 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.423144 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.423344 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.423422 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:40.426091 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:40.426094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:40.426097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:40.426119 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:40.427775 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:40.428237 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:40.428240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:40.428241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:40.428310 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:40.428588 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:40.428590 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:40.428591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:40.428753 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:40.428826 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:40.429271 (+ 445us) server_negotiation.cc:299] Negotiation successful
0927 10:46:40.429365 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 10:46:40.430346 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:40.418589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.3:56553)
0927 10:46:40.419239 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:40.419280 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:40.420030 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:40.420506 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:40.420512 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:40.420865 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:40.421326 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:40.421339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.422140 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.422143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:40.423039 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:40.423047 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.423474 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.423477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:40.423610 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:40.424241 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:40.424257 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:40.425915 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:40.427872 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:40.427877 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:40.427889 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:40.428152 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:40.428385 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:40.428387 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:40.428388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:40.428510 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:40.428835 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:40.428840 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:40.429070 (+ 230us) client_negotiation.cc:769] Sending connection context
0927 10:46:40.429224 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 10:46:40.429467 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":112,"threads_started":1}
I20260927 10:46:40.431442 16241 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37323
I20260927 10:46:40.431660 16241 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:40.432091 16241 heartbeater.cc:511] Master 127.0.98.62:37323 requested a full tablet report, sending...
I20260927 10:46:40.433046 15778 ts_manager.cc:194] Registered new tserver with Master: fc6fe33c2104438eb903f1ec0d89f221 (127.0.98.3:36377)
I20260927 10:46:40.433578 15778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56553
I20260927 10:46:40.439376 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:46:40.449517 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:40.441451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44858 (local address 127.0.98.62:37323)
0927 10:46:40.441573 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:40.441577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:40.441754 (+ 177us) server_negotiation.cc:407] Connection header received
0927 10:46:40.441850 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:40.441852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:40.441891 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:40.441970 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:40.443013 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.443631 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.444457 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.444682 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.444760 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:40.446186 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:40.446189 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:40.446191 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:40.446211 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:40.447628 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:40.448082 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:40.448084 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:40.448086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:40.448162 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:40.448577 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:40.448579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:40.448580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:40.448794 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:40.448922 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:40.449175 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 10:46:40.449312 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:46:40.452355 15778 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:46:40.455543 15778 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:46:40.470134 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:40.464904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41650 (local address 127.0.98.1:42477)
0927 10:46:40.465621 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:40.465636 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:40.465656 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:46:40.465722 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:40.465726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:40.465922 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:40.466062 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:46:40.466224 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.467026 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.468869 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.469424 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.469544 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:40.469698 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:40.469858 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 10:46:40.469960 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":90,"threads_started":1}
I20260927 10:46:40.470126 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:40.463898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40283 (local address 127.0.0.1:50982)
0927 10:46:40.464935 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:40.464952 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:40.465057 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:40.465655 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:40.465662 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:40.465688 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:46:40.465837 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:40.465843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.467324 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.467328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:40.468650 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:40.468660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.469776 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.469783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:40.469867 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:40.469892 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:46:40.469956 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 10:46:40.470006 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":45,"thread_start_us":94,"threads_started":1}
I20260927 10:46:40.470199 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:40.464106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36377 (local address 127.0.0.1:45708)
0927 10:46:40.464576 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:40.464631 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:40.464757 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:40.465812 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:40.465815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:40.465846 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:46:40.465967 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:40.465975 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.467396 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.467400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:40.468549 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:40.468558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.469800 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.469803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:40.469898 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:40.469929 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 10:46:40.470000 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 10:46:40.470066 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":37,"thread_start_us":92,"threads_started":1}
I20260927 10:46:40.470126 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:40.464597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42477 (local address 127.0.0.1:41650)
0927 10:46:40.465238 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:40.465251 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:40.465333 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:40.466036 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:40.466039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:40.466054 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:46:40.466147 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:40.466153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.467151 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.467158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:40.468747 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:40.468756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.469555 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.469558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:40.469669 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:40.469706 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 10:46:40.469819 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 10:46:40.469962 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":33,"thread_start_us":105,"threads_started":1}
I20260927 10:46:40.470719 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:40.464646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45708 (local address 127.0.98.3:36377)
0927 10:46:40.465152 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:40.465163 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:40.465183 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:46:40.465246 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:40.465251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:40.465456 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:40.465648 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:46:40.466101 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.467249 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.468692 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.469501 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.469606 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:40.469732 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:40.470495 (+ 763us) server_negotiation.cc:299] Negotiation successful
0927 10:46:40.470585 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":137,"threads_started":1}
I20260927 10:46:40.470738 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:40.464088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50982 (local address 127.0.98.2:40283)
0927 10:46:40.464419 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:40.464435 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:40.465160 (+ 725us) server_negotiation.cc:407] Connection header received
0927 10:46:40.465246 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:40.465251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:40.465462 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:40.465648 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:46:40.465963 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.467187 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.468754 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.469511 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.469606 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:40.469733 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:40.470527 (+ 794us) server_negotiation.cc:299] Negotiation successful
0927 10:46:40.470630 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":105,"threads_started":1}
I20260927 10:46:40.473385 15907 tablet_service.cc:1467] Processing CreateTablet for tablet f6075622582b4962996c9c1e0e51426e (DEFAULT_TABLE table=test-table [id=2628ce3062bc481093bcfa7468ea160e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:46:40.473572 16175 tablet_service.cc:1467] Processing CreateTablet for tablet f6075622582b4962996c9c1e0e51426e (DEFAULT_TABLE table=test-table [id=2628ce3062bc481093bcfa7468ea160e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:46:40.473572 16041 tablet_service.cc:1467] Processing CreateTablet for tablet f6075622582b4962996c9c1e0e51426e (DEFAULT_TABLE table=test-table [id=2628ce3062bc481093bcfa7468ea160e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:46:40.474575 15907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6075622582b4962996c9c1e0e51426e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:46:40.474587 16041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6075622582b4962996c9c1e0e51426e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:46:40.474591 16175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6075622582b4962996c9c1e0e51426e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:46:40.480404 16258 tablet_bootstrap.cc:492] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f: Bootstrap starting.
I20260927 10:46:40.482189 16259 tablet_bootstrap.cc:492] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927: Bootstrap starting.
I20260927 10:46:40.482637 16260 tablet_bootstrap.cc:492] T f6075622582b4962996c9c1e0e51426e P fc6fe33c2104438eb903f1ec0d89f221: Bootstrap starting.
I20260927 10:46:40.482981 16258 tablet_bootstrap.cc:654] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f: Neither blocks nor log segments found. Creating new log.
I20260927 10:46:40.484022 16258 log.cc:824] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:40.484503 16260 tablet_bootstrap.cc:654] T f6075622582b4962996c9c1e0e51426e P fc6fe33c2104438eb903f1ec0d89f221: Neither blocks nor log segments found. Creating new log.
I20260927 10:46:40.484879 16259 tablet_bootstrap.cc:654] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927: Neither blocks nor log segments found. Creating new log.
I20260927 10:46:40.485759 16260 log.cc:824] T f6075622582b4962996c9c1e0e51426e P fc6fe33c2104438eb903f1ec0d89f221: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:40.485759 16259 log.cc:824] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:40.485800 16258 tablet_bootstrap.cc:492] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f: No bootstrap required, opened a new log
I20260927 10:46:40.486027 16258 ts_tablet_manager.cc:1397] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:46:40.487828 16259 tablet_bootstrap.cc:492] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927: No bootstrap required, opened a new log
I20260927 10:46:40.487957 16260 tablet_bootstrap.cc:492] T f6075622582b4962996c9c1e0e51426e P fc6fe33c2104438eb903f1ec0d89f221: No bootstrap required, opened a new log
I20260927 10:46:40.488016 16259 ts_tablet_manager.cc:1397] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:46:40.488085 16260 ts_tablet_manager.cc:1397] T f6075622582b4962996c9c1e0e51426e P fc6fe33c2104438eb903f1ec0d89f221: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:46:40.490060 16258 raft_consensus.cc:348] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:46:40.490322 16258 raft_consensus.cc:374] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:46:40.490373 16258 raft_consensus.cc:729] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5663ccff76448bab7a0118f4d2d315f, State: Initialized, Role: FOLLOWER
I20260927 10:46:40.490906 16258 consensus_queue.cc:260] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:46:40.490862 16260 raft_consensus.cc:348] T f6075622582b4962996c9c1e0e51426e P fc6fe33c2104438eb903f1ec0d89f221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:46:40.491091 16260 raft_consensus.cc:374] T f6075622582b4962996c9c1e0e51426e P fc6fe33c2104438eb903f1ec0d89f221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:46:40.491166 16260 raft_consensus.cc:729] T f6075622582b4962996c9c1e0e51426e P fc6fe33c2104438eb903f1ec0d89f221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc6fe33c2104438eb903f1ec0d89f221, State: Initialized, Role: FOLLOWER
I20260927 10:46:40.491644 15973 heartbeater.cc:503] Master 127.0.98.62:37323 was elected leader, sending a full tablet report...
I20260927 10:46:40.491607 16260 consensus_queue.cc:260] T f6075622582b4962996c9c1e0e51426e P fc6fe33c2104438eb903f1ec0d89f221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:46:40.491950 16258 ts_tablet_manager.cc:1428] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260927 10:46:40.491859 16259 raft_consensus.cc:348] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:46:40.492161 16259 raft_consensus.cc:374] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:46:40.492240 16259 raft_consensus.cc:729] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e22505803753467296b25b5d6ea2f927, State: Initialized, Role: FOLLOWER
I20260927 10:46:40.492504 16260 ts_tablet_manager.cc:1428] T f6075622582b4962996c9c1e0e51426e P fc6fe33c2104438eb903f1ec0d89f221: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 10:46:40.492659 16259 consensus_queue.cc:260] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:46:40.493048 16241 heartbeater.cc:503] Master 127.0.98.62:37323 was elected leader, sending a full tablet report...
I20260927 10:46:40.493546 16259 ts_tablet_manager.cc:1428] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:46:40.493981 16107 heartbeater.cc:503] Master 127.0.98.62:37323 was elected leader, sending a full tablet report...
I20260927 10:46:40.528414 16266 raft_consensus.cc:482] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:46:40.528656 16266 raft_consensus.cc:504] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:46:40.529886 16266 leader_election.cc:288] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc6fe33c2104438eb903f1ec0d89f221 (127.0.98.3:36377), a5663ccff76448bab7a0118f4d2d315f (127.0.98.1:42477)
I20260927 10:46:40.534926 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:40.530231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46039 (local address 127.0.98.3:36377)
0927 10:46:40.530357 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:40.530361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:40.530925 (+ 564us) server_negotiation.cc:407] Connection header received
0927 10:46:40.531028 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:40.531031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:40.531082 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:40.531139 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:46:40.531463 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.532076 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.533447 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.534042 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.534135 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:40.534192 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:40.534709 (+ 517us) server_negotiation.cc:299] Negotiation successful
0927 10:46:40.534788 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:46:40.534935 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:40.530278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42439 (local address 127.0.98.1:42477)
0927 10:46:40.530435 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:40.530440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:40.530456 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:46:40.530597 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:40.530601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:40.530647 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:40.530742 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:46:40.531060 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.531716 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.533362 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.534004 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.534126 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:40.534192 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:40.534709 (+ 517us) server_negotiation.cc:299] Negotiation successful
0927 10:46:40.534788 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:46:40.534982 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:40.530167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42477 (local address 127.0.98.2:42439)
0927 10:46:40.530304 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:40.530321 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:40.530464 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:40.530765 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:40.530768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:40.530785 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:46:40.530923 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:40.530930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.531822 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.531828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:40.533164 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:40.533175 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.534216 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.534220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:40.534508 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:40.534560 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 10:46:40.534652 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 10:46:40.534796 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":137}
I20260927 10:46:40.534982 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:40.530168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36377 (local address 127.0.98.2:46039)
0927 10:46:40.530834 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:40.530845 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:40.530931 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:40.531171 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:40.531174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:40.531191 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:46:40.531302 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:40.531308 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.532221 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.532224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:40.533336 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:40.533342 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.534216 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.534220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:40.534503 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:40.534562 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 10:46:40.534652 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 10:46:40.534796 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":432,"thread_start_us":162,"threads_started":1}
I20260927 10:46:40.535888 15927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6075622582b4962996c9c1e0e51426e" candidate_uuid: "e22505803753467296b25b5d6ea2f927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5663ccff76448bab7a0118f4d2d315f" is_pre_election: true
I20260927 10:46:40.535892 16195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6075622582b4962996c9c1e0e51426e" candidate_uuid: "e22505803753467296b25b5d6ea2f927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc6fe33c2104438eb903f1ec0d89f221" is_pre_election: true
I20260927 10:46:40.536224 16195 raft_consensus.cc:2454] T f6075622582b4962996c9c1e0e51426e P fc6fe33c2104438eb903f1ec0d89f221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e22505803753467296b25b5d6ea2f927 in term 0.
I20260927 10:46:40.536224 15927 raft_consensus.cc:2454] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e22505803753467296b25b5d6ea2f927 in term 0.
I20260927 10:46:40.536772 15994 leader_election.cc:302] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a5663ccff76448bab7a0118f4d2d315f, e22505803753467296b25b5d6ea2f927, fc6fe33c2104438eb903f1ec0d89f221; no voters:
I20260927 10:46:40.537093 16266 raft_consensus.cc:2781] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:46:40.537184 16266 raft_consensus.cc:482] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:46:40.537256 16266 raft_consensus.cc:3037] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:46:40.538551 16266 raft_consensus.cc:504] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:46:40.538957 16266 leader_election.cc:288] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927 [CANDIDATE]: Term 1 election: Requested vote from peers fc6fe33c2104438eb903f1ec0d89f221 (127.0.98.3:36377), a5663ccff76448bab7a0118f4d2d315f (127.0.98.1:42477)
I20260927 10:46:40.539764 15927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6075622582b4962996c9c1e0e51426e" candidate_uuid: "e22505803753467296b25b5d6ea2f927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5663ccff76448bab7a0118f4d2d315f"
I20260927 10:46:40.539757 16195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6075622582b4962996c9c1e0e51426e" candidate_uuid: "e22505803753467296b25b5d6ea2f927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc6fe33c2104438eb903f1ec0d89f221"
I20260927 10:46:40.539912 15927 raft_consensus.cc:3037] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:46:40.539948 16195 raft_consensus.cc:3037] T f6075622582b4962996c9c1e0e51426e P fc6fe33c2104438eb903f1ec0d89f221 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:46:40.541232 16195 raft_consensus.cc:2454] T f6075622582b4962996c9c1e0e51426e P fc6fe33c2104438eb903f1ec0d89f221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e22505803753467296b25b5d6ea2f927 in term 1.
I20260927 10:46:40.541232 15927 raft_consensus.cc:2454] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e22505803753467296b25b5d6ea2f927 in term 1.
I20260927 10:46:40.541637 15994 leader_election.cc:302] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a5663ccff76448bab7a0118f4d2d315f, e22505803753467296b25b5d6ea2f927, fc6fe33c2104438eb903f1ec0d89f221; no voters:
I20260927 10:46:40.541877 16266 raft_consensus.cc:2781] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:46:40.542068 16266 raft_consensus.cc:686] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927 [term 1 LEADER]: Becoming Leader. State: Replica: e22505803753467296b25b5d6ea2f927, State: Running, Role: LEADER
I20260927 10:46:40.542459 16266 consensus_queue.cc:237] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:46:40.545568 15781 catalog_manager.cc:5495] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927 reported cstate change: term changed from 0 to 1, leader changed from <none> to e22505803753467296b25b5d6ea2f927 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e22505803753467296b25b5d6ea2f927" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } health_report { overall_health: UNKNOWN } } }
I20260927 10:46:40.557554 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:40.552937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50992 (local address 127.0.98.2:40283)
0927 10:46:40.553057 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:40.553061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:40.553074 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:46:40.553391 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:40.553395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:40.553494 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:40.553645 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:46:40.553919 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.554792 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.555759 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:40.556048 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:40.556176 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:40.556528 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:46:40.556971 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:46:40.557082 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:40.557313 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 10:46:40.557404 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:46:40.563740 15927 raft_consensus.cc:1261] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f [term 1 FOLLOWER]: Refusing update from remote peer e22505803753467296b25b5d6ea2f927: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:46:40.563808 16195 raft_consensus.cc:1261] T f6075622582b4962996c9c1e0e51426e P fc6fe33c2104438eb903f1ec0d89f221 [term 1 FOLLOWER]: Refusing update from remote peer e22505803753467296b25b5d6ea2f927: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:46:40.564648 16268 consensus_queue.cc:1035] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:46:40.565009 16266 consensus_queue.cc:1035] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:46:40.576620 16273 mvcc.cc:205] Tried to move back new op lower bound from 7333912578301440000 to 7333912578224640000. Current Snapshot: MvccSnapshot[applied={T|T < 7333912578301440000 or (T in {7333912578301440000})}]
I20260927 10:46:40.581781 15781 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:46:40.581980 15781 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:46:40.584702 15781 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73: Sending DeleteTablet for 3 replicas of tablet f6075622582b4962996c9c1e0e51426e
I20260927 10:46:40.585561 16041 tablet_service.cc:1514] Processing DeleteTablet for tablet f6075622582b4962996c9c1e0e51426e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:46:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50982
I20260927 10:46:40.585621 16175 tablet_service.cc:1514] Processing DeleteTablet for tablet f6075622582b4962996c9c1e0e51426e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:46:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45708
I20260927 10:46:40.585724 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 15844
I20260927 10:46:40.586709 16281 tablet_replica.cc:330] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927: stopping tablet replica
I20260927 10:46:40.587101 16281 raft_consensus.cc:2229] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:46:40.587760 16281 raft_consensus.cc:2258] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:46:40.589388 16281 ts_tablet_manager.cc:1905] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:46:40.593001 16281 ts_tablet_manager.cc:1918] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:46:40.593116 16281 log.cc:1196] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/wal/wals/f6075622582b4962996c9c1e0e51426e
W20260927 10:46:40.593351 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42477 (error 108)
I20260927 10:46:40.593509 16281 ts_tablet_manager.cc:1939] T f6075622582b4962996c9c1e0e51426e P e22505803753467296b25b5d6ea2f927: Deleting consensus metadata
I20260927 10:46:40.593854 16282 tablet_replica.cc:330] T f6075622582b4962996c9c1e0e51426e P fc6fe33c2104438eb903f1ec0d89f221: stopping tablet replica
I20260927 10:46:40.594238 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 15978
I20260927 10:46:40.594247 16282 raft_consensus.cc:2229] T f6075622582b4962996c9c1e0e51426e P fc6fe33c2104438eb903f1ec0d89f221 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:46:40.594576 16282 raft_consensus.cc:2258] T f6075622582b4962996c9c1e0e51426e P fc6fe33c2104438eb903f1ec0d89f221 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:46:40.595722 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:40.595321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42477 (local address 127.0.0.1:41652)
0927 10:46:40.595536 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:40.595614 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260927 10:46:40.596139 15766 catalog_manager.cc:4584] TS a5663ccff76448bab7a0118f4d2d315f (127.0.98.1:42477): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f6075622582b4962996c9c1e0e51426e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42477: connect: Connection refused (error 111)
I20260927 10:46:40.596791 16282 ts_tablet_manager.cc:1905] T f6075622582b4962996c9c1e0e51426e P fc6fe33c2104438eb903f1ec0d89f221: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:46:40.600188 16282 ts_tablet_manager.cc:1918] T f6075622582b4962996c9c1e0e51426e P fc6fe33c2104438eb903f1ec0d89f221: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:46:40.600298 16282 log.cc:1196] T f6075622582b4962996c9c1e0e51426e P fc6fe33c2104438eb903f1ec0d89f221: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/f6075622582b4962996c9c1e0e51426e
I20260927 10:46:40.600646 16282 ts_tablet_manager.cc:1939] T f6075622582b4962996c9c1e0e51426e P fc6fe33c2104438eb903f1ec0d89f221: Deleting consensus metadata
I20260927 10:46:40.601797 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 16112
I20260927 10:46:40.603193 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:40.602117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40283 (local address 127.0.0.1:50994)
0927 10:46:40.603020 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:40.603089 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":793}
I20260927 10:46:40.607710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 15750
I20260927 10:47:00.614293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37323
--webserver_interface=127.0.98.62
--webserver_port=42739
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:00.728585 16284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:00.728876 16284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:00.728917 16284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:00.733210 16284 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 10:47:00.733285 16284 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 10:47:00.733302 16284 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 10:47:00.733316 16284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:00.733336 16284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:00.733353 16284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:00.733419 16284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:00.733453 16284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:00.738991 16284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:00.740180 16284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:00.741268 16284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:00.754767 16291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:00.754715 16290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:00.754879 16284 server_base.cc:1034] running on GCE node
W20260927 10:47:00.754949 16293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:00.755779 16284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:00.756990 16284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:00.758227 16284 hybrid_clock.cc:648] HybridClock initialized: now 1790506020758207 us; error 44 us; skew 500 ppm
Sep 27 10:47:00 dist-test-slave-dzhw krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:00.769625 16284 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:47:00.770890 16284 webserver.cc:460] Webserver started at http://127.0.98.62:42739/ using document root <none> and password file <none>
I20260927 10:47:00.771775 16284 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:00.771895 16284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:00.776082 16284 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:47:00.778541 16300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:00.779904 16284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:47:00.780078 16284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "52f42c39e2e74738bed67e4222acad73"
format_stamp: "Formatted at 2026-09-27 10:46:39 on dist-test-slave-dzhw"
server_key: "2cd0c87e967318a860efbe877a7ce38b"
server_key_iv: "b77921142fad963c2fd504e7a49e8a9c"
server_key_version: "encryptionkey@0"
I20260927 10:47:00.780737 16284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:00.806929 16284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:00.810451 16284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:00.810679 16284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:00.818174 16284 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37323
I20260927 10:47:00.818580 16352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37323 every 8 connection(s)
I20260927 10:47:00.819357 16284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:47:00.821671 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 16284
I20260927 10:47:00.822153 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42477
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37323
--webserver_port=36605
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:47:00.824522 16353 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:47:00.826383 16353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73: Bootstrap starting.
I20260927 10:47:00.843971 16353 log.cc:824] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:00.852965 16353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:47:00.853407 16353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73: Bootstrap complete.
I20260927 10:47:00.857806 16353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f42c39e2e74738bed67e4222acad73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37323 } }
I20260927 10:47:00.858222 16353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f42c39e2e74738bed67e4222acad73, State: Initialized, Role: FOLLOWER
I20260927 10:47:00.858770 16353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f42c39e2e74738bed67e4222acad73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37323 } }
I20260927 10:47:00.858887 16353 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:47:00.858995 16353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:47:00.859102 16353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:47:00.861235 16353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f42c39e2e74738bed67e4222acad73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37323 } }
I20260927 10:47:00.861603 16353 leader_election.cc:302] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52f42c39e2e74738bed67e4222acad73; no voters:
I20260927 10:47:00.861967 16353 leader_election.cc:288] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 10:47:00.862062 16357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 10:47:00.862318 16357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [term 2 LEADER]: Becoming Leader. State: Replica: 52f42c39e2e74738bed67e4222acad73, State: Running, Role: LEADER
I20260927 10:47:00.862696 16357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f42c39e2e74738bed67e4222acad73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37323 } }
I20260927 10:47:00.863282 16353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:47:00.864555 16358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "52f42c39e2e74738bed67e4222acad73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f42c39e2e74738bed67e4222acad73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37323 } } }
I20260927 10:47:00.864702 16358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:00.865046 16359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52f42c39e2e74738bed67e4222acad73. Latest consensus state: current_term: 2 leader_uuid: "52f42c39e2e74738bed67e4222acad73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f42c39e2e74738bed67e4222acad73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37323 } } }
I20260927 10:47:00.865140 16359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:00.865525 16366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:47:00.871353 16366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:47:00.872026 16366 catalog_manager.cc:1259] Loaded cluster ID: 8bd3389dea394e959f1a07f02b555330
I20260927 10:47:00.872093 16366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:47:00.874775 16366 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:47:00.875519 16366 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73: Loaded TSK: 0
I20260927 10:47:00.891394 16366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73: Generated new TSK 1
I20260927 10:47:00.891723 16366 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:47:00.949182 16355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:00.949450 16355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:00.949496 16355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:00.953655 16355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:00.953725 16355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:00.953786 16355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:47:00.959975 16355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36605
--tserver_master_addrs=127.0.98.62:37323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:00.961069 16355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:00.962100 16355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:00.972366 16383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:00.972371 16381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:00.972366 16380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:00.972565 16355 server_base.cc:1034] running on GCE node
I20260927 10:47:00.973171 16355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:00.973922 16355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:00.975126 16355 hybrid_clock.cc:648] HybridClock initialized: now 1790506020975103 us; error 40 us; skew 500 ppm
Sep 27 10:47:00 dist-test-slave-dzhw krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:00.977994 16355 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 10:47:00.979079 16355 webserver.cc:460] Webserver started at http://127.0.98.1:36605/ using document root <none> and password file <none>
I20260927 10:47:00.979667 16355 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:00.979740 16355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:00.983954 16355 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:47:00.986317 16390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:00.987690 16355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:47:00.987790 16355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "a5663ccff76448bab7a0118f4d2d315f"
format_stamp: "Formatted at 2026-09-27 10:46:39 on dist-test-slave-dzhw"
server_key: "c23aa5f65c6d5336923cfd3a4519e335"
server_key_iv: "25130d513879fb8f7a78f07e1dc1b4f4"
server_key_version: "encryptionkey@0"
I20260927 10:47:00.988375 16355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:01.003997 16355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:01.007444 16355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:01.007658 16355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:01.008349 16355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:01.009835 16397 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 10:47:01.015758 16355 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 10:47:01.015883 16355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260927 10:47:01.015962 16355 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 10:47:01.017803 16355 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 10:47:01.017889 16355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 10:47:01.018217 16397 tablet_bootstrap.cc:492] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f: Bootstrap starting.
I20260927 10:47:01.027729 16355 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42477
I20260927 10:47:01.027756 16504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42477 every 8 connection(s)
I20260927 10:47:01.028739 16355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:47:01.029724 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 16355
I20260927 10:47:01.030180 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40283
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37323
--webserver_port=33287
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:47:01.040485 16397 log.cc:824] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:47:01 dist-test-slave-dzhw krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:47:01.048728 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:01.031860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43931 (local address 127.0.98.62:37323)
0927 10:47:01.033045 (+ 1185us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:01.033060 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:01.033473 (+ 413us) server_negotiation.cc:407] Connection header received
0927 10:47:01.034683 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:01.034698 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:01.035067 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:01.035425 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:01.036570 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.037547 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.038839 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.039416 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.039612 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:01.042805 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:01.042812 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:01.042828 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:01.042863 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:01.045061 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:01.045739 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:01.045745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:01.045750 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:01.045901 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:01.046324 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:01.046327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:01.046329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:01.046835 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:01.046969 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:01.047610 (+ 641us) server_negotiation.cc:299] Negotiation successful
0927 10:47:01.047903 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1049,"thread_start_us":125,"threads_started":1}
I20260927 10:47:01.048766 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:01.031155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.1:43931)
0927 10:47:01.033281 (+ 2126us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:01.033324 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:01.034188 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:01.035473 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:01.035481 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:01.035923 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:01.036383 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:01.036397 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.037725 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.037735 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:01.038698 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:01.038708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.039575 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.039580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:01.039786 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:01.040521 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:01.040538 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:01.042563 (+ 2025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:01.045194 (+ 2631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:01.045207 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:01.045222 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:01.045584 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:01.046020 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:01.046023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:01.046025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:01.046192 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:01.046955 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:01.046961 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:01.047265 (+ 304us) client_negotiation.cc:769] Sending connection context
0927 10:47:01.047461 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 10:47:01.047802 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1929,"thread_start_us":137,"threads_started":1}
I20260927 10:47:01.049328 16397 tablet_bootstrap.cc:492] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:47:01.049790 16397 tablet_bootstrap.cc:492] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f: Bootstrap complete.
I20260927 10:47:01.051046 16397 ts_tablet_manager.cc:1397] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.007s
I20260927 10:47:01.051826 16505 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37323
I20260927 10:47:01.052045 16505 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:01.052592 16505 heartbeater.cc:511] Master 127.0.98.62:37323 requested a full tablet report, sending...
I20260927 10:47:01.053459 16397 raft_consensus.cc:348] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:47:01.053889 16397 raft_consensus.cc:729] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5663ccff76448bab7a0118f4d2d315f, State: Initialized, Role: FOLLOWER
I20260927 10:47:01.054368 16397 consensus_queue.cc:260] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:47:01.055365 16397 ts_tablet_manager.cc:1428] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 10:47:01.055383 16317 ts_manager.cc:194] Registered new tserver with Master: a5663ccff76448bab7a0118f4d2d315f (127.0.98.1:42477)
I20260927 10:47:01.059312 16317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43931
I20260927 10:47:01.061996 16505 heartbeater.cc:503] Master 127.0.98.62:37323 was elected leader, sending a full tablet report...
Sep 27 10:47:01 dist-test-slave-dzhw krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 10:47:01.070020 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:01.058879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47594 (local address 127.0.98.1:42477)
0927 10:47:01.059592 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:01.059601 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:01.059942 (+ 341us) server_negotiation.cc:407] Connection header received
0927 10:47:01.060201 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:01.060206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:01.060399 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:01.060546 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:01.061152 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.061873 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.062985 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.063438 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.063562 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:01.066186 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:01.066194 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:01.066204 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:01.066226 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:01.067871 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:01.068457 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:01.068463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:01.068464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:01.068558 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:01.068958 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:01.068961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:01.068962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:01.069214 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:01.069338 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:01.069748 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 10:47:01.069895 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":112,"threads_started":1}
I20260927 10:47:01.070025 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:01.058849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42477 (local address 127.0.0.1:47594)
0927 10:47:01.059806 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:01.059840 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:01.060011 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:01.060551 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:01.060557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:01.060915 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:01.061014 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:01.061021 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.062049 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.062052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:01.062857 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:01.062867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.063549 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.063565 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:01.063679 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:01.064299 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:01.064314 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:01.066025 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:01.067990 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:01.067999 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:01.068004 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:01.068309 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:01.068731 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:01.068734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:01.068736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:01.068834 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:01.069339 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:01.069345 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:01.069485 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 10:47:01.069635 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 10:47:01.069836 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":134,"threads_started":1}
I20260927 10:47:01.071081 16439 tablet_service.cc:1514] Processing DeleteTablet for tablet f6075622582b4962996c9c1e0e51426e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:46:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47594
I20260927 10:47:01.071931 16514 tablet_replica.cc:330] stopping tablet replica
I20260927 10:47:01.072224 16514 raft_consensus.cc:2229] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:47:01.072479 16514 raft_consensus.cc:2258] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:01.073433 16514 ts_tablet_manager.cc:1905] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:47:01.077355 16514 ts_tablet_manager.cc:1918] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:47:01.077445 16514 log.cc:1196] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/wal/wals/f6075622582b4962996c9c1e0e51426e
I20260927 10:47:01.077711 16514 ts_tablet_manager.cc:1939] T f6075622582b4962996c9c1e0e51426e P a5663ccff76448bab7a0118f4d2d315f: Deleting consensus metadata
I20260927 10:47:01.079568 16301 catalog_manager.cc:4841] TS a5663ccff76448bab7a0118f4d2d315f (127.0.98.1:42477): tablet f6075622582b4962996c9c1e0e51426e (table test-table [id=2628ce3062bc481093bcfa7468ea160e]) successfully deleted
W20260927 10:47:01.155551 16507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:01.155875 16507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:01.155958 16507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:01.160670 16507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:01.160804 16507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:01.160907 16507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:47:01.168460 16507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33287
--tserver_master_addrs=127.0.98.62:37323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:01.169613 16507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:01.170701 16507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:01.180905 16519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:01.180833 16520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:01.180833 16522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:01.181236 16507 server_base.cc:1034] running on GCE node
I20260927 10:47:01.181665 16507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:01.182369 16507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:01.183630 16507 hybrid_clock.cc:648] HybridClock initialized: now 1790506021183617 us; error 27 us; skew 500 ppm
Sep 27 10:47:01 dist-test-slave-dzhw krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:01.186887 16507 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 10:47:01.187829 16507 webserver.cc:460] Webserver started at http://127.0.98.2:33287/ using document root <none> and password file <none>
I20260927 10:47:01.188359 16507 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:01.188429 16507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:01.192469 16507 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:47:01.194795 16529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:01.196050 16507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:47:01.196180 16507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "e22505803753467296b25b5d6ea2f927"
format_stamp: "Formatted at 2026-09-27 10:46:40 on dist-test-slave-dzhw"
server_key: "fe145c6006e916b10346244a0eddab0b"
server_key_iv: "cb08fb4a940d25a44887b8fc67054933"
server_key_version: "encryptionkey@0"
I20260927 10:47:01.196731 16507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:01.214854 16507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:01.218648 16507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:01.218880 16507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:01.219508 16507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:01.220362 16507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:01.220409 16507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:01.220448 16507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:01.220464 16507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:01.229584 16507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40283
I20260927 10:47:01.229604 16642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40283 every 8 connection(s)
I20260927 10:47:01.230742 16507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:47:01.231590 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 16507
I20260927 10:47:01.231999 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36377
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37323
--webserver_port=37773
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:47:01 dist-test-slave-dzhw krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:47:01.246577 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:01.232833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46867 (local address 127.0.98.62:37323)
0927 10:47:01.232995 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:01.232999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:01.234130 (+ 1131us) server_negotiation.cc:407] Connection header received
0927 10:47:01.235120 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:01.235123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:01.235179 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:01.235270 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:01.236628 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.237581 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.238862 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.239333 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.239468 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:01.242601 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:01.242609 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:01.242614 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:01.242654 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:01.244389 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:01.244928 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:01.244932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:01.244934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:01.245019 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:01.245346 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:01.245349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:01.245351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:01.245577 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:01.245690 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:01.246228 (+ 538us) server_negotiation.cc:299] Negotiation successful
0927 10:47:01.246367 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 10:47:01.247422 16646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:01.233274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.2:46867)
0927 10:47:01.233965 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:01.234006 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:01.234909 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:01.235518 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:01.235525 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:01.235884 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:01.236430 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:01.236452 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.237720 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.237723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:01.238726 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:01.238736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.239509 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.239512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:01.239714 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:01.240429 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:01.240445 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:01.242357 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:01.244513 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:01.244519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:01.244532 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:01.244818 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:01.245107 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:01.245109 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:01.245110 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:01.245246 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:01.245696 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:01.245701 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:01.245961 (+ 260us) client_negotiation.cc:769] Sending connection context
0927 10:47:01.246139 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 10:47:01.246405 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":156,"threads_started":1}
I20260927 10:47:01.248726 16643 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37323
I20260927 10:47:01.248998 16643 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:01.249544 16643 heartbeater.cc:511] Master 127.0.98.62:37323 requested a full tablet report, sending...
I20260927 10:47:01.250689 16317 ts_manager.cc:194] Registered new tserver with Master: e22505803753467296b25b5d6ea2f927 (127.0.98.2:40283)
I20260927 10:47:01.251320 16317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46867
W20260927 10:47:01.349325 16645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:01.349643 16645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:01.349723 16645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:01.353907 16645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:01.354007 16645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:01.354094 16645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:47:01.359474 16645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37773
--tserver_master_addrs=127.0.98.62:37323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:01.360610 16645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:01.361673 16645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:01.371973 16651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:01.372124 16654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:01.372159 16645 server_base.cc:1034] running on GCE node
W20260927 10:47:01.372393 16652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:01.372838 16645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:01.373493 16645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:01.374681 16645 hybrid_clock.cc:648] HybridClock initialized: now 1790506021374668 us; error 34 us; skew 500 ppm
Sep 27 10:47:01 dist-test-slave-dzhw krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:01.377519 16645 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 10:47:01.378386 16645 webserver.cc:460] Webserver started at http://127.0.98.3:37773/ using document root <none> and password file <none>
I20260927 10:47:01.379052 16645 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:01.379155 16645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:01.382947 16645 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 10:47:01.385286 16661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:01.386698 16645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:47:01.386870 16645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "fc6fe33c2104438eb903f1ec0d89f221"
format_stamp: "Formatted at 2026-09-27 10:46:40 on dist-test-slave-dzhw"
server_key: "c291b54b396fe0e92a0e1fab372a57a7"
server_key_iv: "dc55ad3b0ed46668b4300a51129b235b"
server_key_version: "encryptionkey@0"
I20260927 10:47:01.387439 16645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:01.402292 16645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:01.405743 16645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:01.405977 16645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:01.406661 16645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:01.407510 16645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:01.407588 16645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:01.407657 16645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:01.407692 16645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:01.417104 16645 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36377
I20260927 10:47:01.417128 16774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36377 every 8 connection(s)
I20260927 10:47:01.418138 16645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:47:01.419680 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 16645
Sep 27 10:47:01 dist-test-slave-dzhw krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:47:01.429136 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:01.420404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40826 (local address 127.0.98.62:37323)
0927 10:47:01.420648 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:01.420652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:01.420863 (+ 211us) server_negotiation.cc:407] Connection header received
0927 10:47:01.420958 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:01.420965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:01.421025 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:01.421128 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:01.422069 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.422707 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.423918 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.424142 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.424233 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:01.425724 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:01.425728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:01.425730 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:01.425761 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:01.427415 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:01.427873 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:01.427878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:01.427880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:01.427964 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:01.428258 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:01.428261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:01.428263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:01.428461 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:01.428661 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:01.428840 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 10:47:01.428961 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
I20260927 10:47:01.432891 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:01.420137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42801 (local address 127.0.98.62:37323)
0927 10:47:01.420321 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:01.420326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:01.421756 (+ 1430us) server_negotiation.cc:407] Connection header received
0927 10:47:01.422794 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:01.422799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:01.422853 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:01.422928 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:01.424373 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.424840 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.426063 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.426386 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.426524 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:01.428993 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:01.428997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:01.429002 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:01.429059 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:01.430712 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:01.431343 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:01.431346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:01.431348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:01.431437 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:01.431796 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:01.431799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:01.431801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:01.431974 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:01.432084 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:01.432605 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 10:47:01.432719 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 10:47:01.433760 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:01.420628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.3:42801)
0927 10:47:01.421565 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:01.421627 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:01.422597 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:01.423130 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:01.423139 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:01.423611 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:01.424180 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:01.424226 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.424983 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.424986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:01.425911 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:01.425921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.426597 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.426601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:01.426842 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:01.427414 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:01.427430 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:01.428818 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:01.430849 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:01.430854 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:01.430871 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:01.431217 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:01.431532 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:01.431536 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:01.431538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:01.431682 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:01.432083 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:01.432092 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:01.432357 (+ 265us) client_negotiation.cc:769] Sending connection context
0927 10:47:01.432553 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 10:47:01.432877 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":178,"threads_started":1}
I20260927 10:47:01.434991 16775 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37323
I20260927 10:47:01.435235 16775 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:01.435746 16775 heartbeater.cc:511] Master 127.0.98.62:37323 requested a full tablet report, sending...
I20260927 10:47:01.436836 16317 ts_manager.cc:194] Registered new tserver with Master: fc6fe33c2104438eb903f1ec0d89f221 (127.0.98.3:36377)
I20260927 10:47:01.437379 16317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42801
I20260927 10:47:01.443446 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 10:47:01.448601 16247 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:37323)
I20260927 10:47:01.449100 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:01.444105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40828 (local address 127.0.98.62:37323)
0927 10:47:01.444283 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:01.444288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:01.444470 (+ 182us) server_negotiation.cc:407] Connection header received
0927 10:47:01.444675 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:01.444678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:01.444830 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:01.444954 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:47:01.445192 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.445723 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.446706 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.446919 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.447037 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:01.447363 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:47:01.447506 (+ 143us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 10:47:01.448920 (+ 1414us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40828: token expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 10:47:01.449174 16509 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40828: token expired
I20260927 10:47:01.457353 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:01.449060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40844 (local address 127.0.98.62:37323)
0927 10:47:01.449171 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:01.449176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:01.449547 (+ 371us) server_negotiation.cc:407] Connection header received
0927 10:47:01.449659 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:01.449663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:01.449709 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:01.449805 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:01.450526 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.451142 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.451976 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.452421 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.452545 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:01.454158 (+ 1613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:01.454161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:01.454163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:01.454188 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:01.455764 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:01.456150 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:01.456152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:01.456154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:01.456227 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:01.456462 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:01.456464 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:01.456465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:01.456635 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:01.456738 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:01.457017 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 10:47:01.457120 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 10:47:01.459949 16317 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:47:01.463326 16317 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:47:01.472371 16439 tablet_service.cc:1467] Processing CreateTablet for tablet 6670e2917f674a4c83f16b4105e332d8 (DEFAULT_TABLE table=test-table [id=cc8baef6758543ed96855d2ccd1ca5de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:01.473047 16439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6670e2917f674a4c83f16b4105e332d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:01.475821 16397 tablet_bootstrap.cc:492] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f: Bootstrap starting.
I20260927 10:47:01.476818 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:01.471035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36377 (local address 127.0.0.1:46482)
0927 10:47:01.471173 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:01.471193 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:01.471267 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:01.472226 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:01.472229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:01.472260 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:01.472378 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:01.472387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.474004 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.474007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:01.475215 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:01.475225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.476244 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.476248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:01.476356 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:01.476458 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 10:47:01.476535 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 10:47:01.476658 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":59}
I20260927 10:47:01.476886 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:01.471262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46482 (local address 127.0.98.3:36377)
0927 10:47:01.471655 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:01.471668 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:01.471692 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:47:01.471811 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:01.471817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:01.472054 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:01.472222 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:01.472484 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.473873 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.475352 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.476111 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.476446 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:01.476586 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:01.476640 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 10:47:01.476727 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":168,"threads_started":1}
I20260927 10:47:01.477034 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:01.471632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41200 (local address 127.0.98.2:40283)
0927 10:47:01.472010 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:01.472021 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:01.472041 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:47:01.472113 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:01.472120 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:01.472304 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:01.472444 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:01.472863 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.473956 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.475132 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.475983 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.476095 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:01.476275 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:01.476791 (+ 516us) server_negotiation.cc:299] Negotiation successful
0927 10:47:01.476879 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":118,"threads_started":1}
I20260927 10:47:01.477097 16397 tablet_bootstrap.cc:654] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:01.477283 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:01.471426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40283 (local address 127.0.0.1:41200)
0927 10:47:01.471736 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:01.471753 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:01.471835 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:01.472622 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:01.472625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:01.472644 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:01.472728 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:01.472736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.474055 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.474059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:01.474999 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:01.475008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.476328 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.476331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:01.476426 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:01.476509 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 10:47:01.476584 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 10:47:01.476664 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":83,"thread_start_us":91,"threads_started":1}
I20260927 10:47:01.479212 16397 tablet_bootstrap.cc:492] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f: No bootstrap required, opened a new log
I20260927 10:47:01.479458 16397 ts_tablet_manager.cc:1397] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 10:47:01.479910 16397 raft_consensus.cc:348] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:47:01.480046 16397 raft_consensus.cc:374] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:01.480093 16397 raft_consensus.cc:729] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5663ccff76448bab7a0118f4d2d315f, State: Initialized, Role: FOLLOWER
I20260927 10:47:01.480226 16397 consensus_queue.cc:260] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:47:01.479964 16577 tablet_service.cc:1467] Processing CreateTablet for tablet 6670e2917f674a4c83f16b4105e332d8 (DEFAULT_TABLE table=test-table [id=cc8baef6758543ed96855d2ccd1ca5de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:01.479959 16709 tablet_service.cc:1467] Processing CreateTablet for tablet 6670e2917f674a4c83f16b4105e332d8 (DEFAULT_TABLE table=test-table [id=cc8baef6758543ed96855d2ccd1ca5de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:01.480547 16397 ts_tablet_manager.cc:1428] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:47:01.481088 16577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6670e2917f674a4c83f16b4105e332d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:01.481093 16709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6670e2917f674a4c83f16b4105e332d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:01.489140 16788 tablet_bootstrap.cc:492] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221: Bootstrap starting.
I20260927 10:47:01.490127 16789 tablet_bootstrap.cc:492] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927: Bootstrap starting.
I20260927 10:47:01.491530 16788 tablet_bootstrap.cc:654] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:01.492631 16789 tablet_bootstrap.cc:654] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:01.492861 16788 log.cc:824] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:01.493340 16789 log.cc:824] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:01.495034 16788 tablet_bootstrap.cc:492] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221: No bootstrap required, opened a new log
I20260927 10:47:01.495224 16789 tablet_bootstrap.cc:492] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927: No bootstrap required, opened a new log
I20260927 10:47:01.495280 16788 ts_tablet_manager.cc:1397] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:47:01.495415 16789 ts_tablet_manager.cc:1397] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:47:01.498311 16789 raft_consensus.cc:348] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:47:01.498311 16788 raft_consensus.cc:348] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:47:01.498661 16789 raft_consensus.cc:374] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:01.498663 16788 raft_consensus.cc:374] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:01.498750 16788 raft_consensus.cc:729] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc6fe33c2104438eb903f1ec0d89f221, State: Initialized, Role: FOLLOWER
I20260927 10:47:01.498721 16789 raft_consensus.cc:729] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e22505803753467296b25b5d6ea2f927, State: Initialized, Role: FOLLOWER
I20260927 10:47:01.499214 16789 consensus_queue.cc:260] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:47:01.499219 16788 consensus_queue.cc:260] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:47:01.500355 16643 heartbeater.cc:503] Master 127.0.98.62:37323 was elected leader, sending a full tablet report...
I20260927 10:47:01.500504 16775 heartbeater.cc:503] Master 127.0.98.62:37323 was elected leader, sending a full tablet report...
I20260927 10:47:01.500340 16789 ts_tablet_manager.cc:1428] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:47:01.500377 16788 ts_tablet_manager.cc:1428] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:47:01.505162 16793 raft_consensus.cc:482] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:47:01.505366 16793 raft_consensus.cc:504] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:47:01.506398 16793 leader_election.cc:288] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e22505803753467296b25b5d6ea2f927 (127.0.98.2:40283), a5663ccff76448bab7a0118f4d2d315f (127.0.98.1:42477)
I20260927 10:47:01.510685 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:01.506763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56605 (local address 127.0.98.2:40283)
0927 10:47:01.506889 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:01.506896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:01.506913 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:47:01.507080 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:01.507082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:01.507155 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:01.507231 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:01.507566 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.508230 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.509539 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.510138 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.510269 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:01.510331 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:01.510536 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 10:47:01.510590 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:47:01.510712 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:01.506705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40283 (local address 127.0.98.3:56605)
0927 10:47:01.506839 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:01.506855 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:01.506945 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:01.507246 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:01.507249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:01.507269 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:01.507414 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:01.507445 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.508334 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.508337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:01.509415 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:01.509424 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.510292 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.510295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:01.510395 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:01.510445 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 10:47:01.510521 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 10:47:01.510588 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 10:47:01.511487 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:01.506963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42477 (local address 127.0.98.3:48699)
0927 10:47:01.507575 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:01.507589 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:01.507685 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:01.507953 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:01.507956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:01.507970 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:01.508085 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:01.508091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.509107 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.509109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:01.510160 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:01.510173 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.511094 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.511097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:01.511184 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:01.511205 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:47:01.511290 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 10:47:01.511350 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":117,"threads_started":1}
I20260927 10:47:01.511533 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:01.507075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48699 (local address 127.0.98.1:42477)
0927 10:47:01.507231 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:01.507235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:01.507687 (+ 452us) server_negotiation.cc:407] Connection header received
0927 10:47:01.507770 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:01.507776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:01.507843 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:01.507943 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:01.508205 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.508994 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.510288 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.510946 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.511051 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:01.511195 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:01.511327 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 10:47:01.511391 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:47:01.511703 16597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6670e2917f674a4c83f16b4105e332d8" candidate_uuid: "fc6fe33c2104438eb903f1ec0d89f221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e22505803753467296b25b5d6ea2f927" is_pre_election: true
I20260927 10:47:01.512032 16597 raft_consensus.cc:2454] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc6fe33c2104438eb903f1ec0d89f221 in term 0.
I20260927 10:47:01.512270 16459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6670e2917f674a4c83f16b4105e332d8" candidate_uuid: "fc6fe33c2104438eb903f1ec0d89f221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5663ccff76448bab7a0118f4d2d315f" is_pre_election: true
I20260927 10:47:01.512542 16459 raft_consensus.cc:2454] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc6fe33c2104438eb903f1ec0d89f221 in term 0.
I20260927 10:47:01.512568 16662 leader_election.cc:302] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e22505803753467296b25b5d6ea2f927, fc6fe33c2104438eb903f1ec0d89f221; no voters:
I20260927 10:47:01.512874 16793 raft_consensus.cc:2781] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:47:01.512961 16793 raft_consensus.cc:482] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:47:01.513006 16793 raft_consensus.cc:3037] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:01.514003 16793 raft_consensus.cc:504] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:47:01.514338 16793 leader_election.cc:288] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 [CANDIDATE]: Term 1 election: Requested vote from peers e22505803753467296b25b5d6ea2f927 (127.0.98.2:40283), a5663ccff76448bab7a0118f4d2d315f (127.0.98.1:42477)
I20260927 10:47:01.514699 16597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6670e2917f674a4c83f16b4105e332d8" candidate_uuid: "fc6fe33c2104438eb903f1ec0d89f221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e22505803753467296b25b5d6ea2f927"
I20260927 10:47:01.514756 16459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6670e2917f674a4c83f16b4105e332d8" candidate_uuid: "fc6fe33c2104438eb903f1ec0d89f221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5663ccff76448bab7a0118f4d2d315f"
I20260927 10:47:01.514835 16597 raft_consensus.cc:3037] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:01.514864 16459 raft_consensus.cc:3037] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:01.516126 16597 raft_consensus.cc:2454] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc6fe33c2104438eb903f1ec0d89f221 in term 1.
I20260927 10:47:01.516126 16459 raft_consensus.cc:2454] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc6fe33c2104438eb903f1ec0d89f221 in term 1.
I20260927 10:47:01.516476 16662 leader_election.cc:302] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e22505803753467296b25b5d6ea2f927, fc6fe33c2104438eb903f1ec0d89f221; no voters:
I20260927 10:47:01.516696 16793 raft_consensus.cc:2781] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:01.516968 16793 raft_consensus.cc:686] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 [term 1 LEADER]: Becoming Leader. State: Replica: fc6fe33c2104438eb903f1ec0d89f221, State: Running, Role: LEADER
I20260927 10:47:01.517302 16793 consensus_queue.cc:237] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:47:01.520666 16316 catalog_manager.cc:5495] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc6fe33c2104438eb903f1ec0d89f221 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "fc6fe33c2104438eb903f1ec0d89f221" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } health_report { overall_health: UNKNOWN } } }
I20260927 10:47:01.534350 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:01.531237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46492 (local address 127.0.98.3:36377)
0927 10:47:01.531363 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:01.531366 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:01.531379 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:47:01.531509 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:01.531512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:01.531564 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:01.531649 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:47:01.531842 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.532325 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.533065 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:01.533263 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:01.533338 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:01.533572 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:47:01.533977 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:47:01.534064 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:01.534118 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 10:47:01.534164 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 10:47:01.535332 16506 tablet.cc:2369] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:47:01.540457 16597 raft_consensus.cc:1261] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927 [term 1 FOLLOWER]: Refusing update from remote peer fc6fe33c2104438eb903f1ec0d89f221: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:47:01.540453 16459 raft_consensus.cc:1261] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [term 1 FOLLOWER]: Refusing update from remote peer fc6fe33c2104438eb903f1ec0d89f221: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:47:01.541208 16795 consensus_queue.cc:1035] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:47:01.541422 16793 consensus_queue.cc:1035] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:47:01.548252 16799 mvcc.cc:205] Tried to move back new op lower bound from 7333912664221818880 to 7333912664137871360. Current Snapshot: MvccSnapshot[applied={T|T < 7333912664221818880}]
I20260927 10:47:01.550890 16800 mvcc.cc:205] Tried to move back new op lower bound from 7333912664221818880 to 7333912664137871360. Current Snapshot: MvccSnapshot[applied={T|T < 7333912664221818880 or (T in {7333912664221818880})}]
I20260927 10:47:01.556282 16315 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:47:01.556452 16315 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:47:01.558964 16315 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73: Sending DeleteTablet for 3 replicas of tablet 6670e2917f674a4c83f16b4105e332d8
I20260927 10:47:01.559788 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 16355
I20260927 10:47:01.559806 16709 tablet_service.cc:1514] Processing DeleteTablet for tablet 6670e2917f674a4c83f16b4105e332d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46482
I20260927 10:47:01.559787 16577 tablet_service.cc:1514] Processing DeleteTablet for tablet 6670e2917f674a4c83f16b4105e332d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41200
I20260927 10:47:01.562673 16808 tablet_replica.cc:330] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221: stopping tablet replica
I20260927 10:47:01.563350 16808 raft_consensus.cc:2229] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:47:01.563866 16808 raft_consensus.cc:2258] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:01.564334 16809 tablet_replica.cc:330] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927: stopping tablet replica
I20260927 10:47:01.564674 16809 raft_consensus.cc:2229] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:47:01.565002 16809 raft_consensus.cc:2258] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:01.565929 16808 ts_tablet_manager.cc:1905] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:47:01.566483 16809 ts_tablet_manager.cc:1905] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 10:47:01.567269 16301 connection.cc:535] client connection to 127.0.98.1:42477 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 10:47:01.567437 16301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 10:47:01.567605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 16507
I20260927 10:47:01.569774 16808 ts_tablet_manager.cc:1918] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:47:01.569883 16808 log.cc:1196] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/6670e2917f674a4c83f16b4105e332d8
I20260927 10:47:01.570209 16808 ts_tablet_manager.cc:1939] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221: Deleting consensus metadata
I20260927 10:47:01.571403 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:01.570992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42477 (local address 127.0.0.1:47608)
0927 10:47:01.571204 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:01.571294 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260927 10:47:01.571419 16302 catalog_manager.cc:4841] TS fc6fe33c2104438eb903f1ec0d89f221 (127.0.98.3:36377): tablet 6670e2917f674a4c83f16b4105e332d8 (table test-table [id=cc8baef6758543ed96855d2ccd1ca5de]) successfully deleted
W20260927 10:47:01.571707 16301 catalog_manager.cc:4584] TS a5663ccff76448bab7a0118f4d2d315f (127.0.98.1:42477): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6670e2917f674a4c83f16b4105e332d8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42477: connect: Connection refused (error 111)
I20260927 10:47:01.575137 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 16645
I20260927 10:47:01.575366 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:01.575081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40283 (local address 127.0.0.1:41216)
0927 10:47:01.575213 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:01.575272 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 10:47:01.580905 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 16284
I20260927 10:47:21.588158 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37323
--webserver_interface=127.0.98.62
--webserver_port=42739
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:21.700125 16811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:21.700460 16811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:21.700547 16811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:21.705322 16811 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 10:47:21.705407 16811 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 10:47:21.705435 16811 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 10:47:21.705458 16811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:21.705554 16811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:21.705617 16811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:21.705658 16811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:21.705693 16811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:21.711181 16811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:21.712275 16811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:21.713299 16811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:21.722468 16816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:21.722535 16819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:21.722510 16811 server_base.cc:1034] running on GCE node
W20260927 10:47:21.722510 16817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:21.723670 16811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:21.724802 16811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:21.725997 16811 hybrid_clock.cc:648] HybridClock initialized: now 1790506041725982 us; error 33 us; skew 500 ppm
Sep 27 10:47:21 dist-test-slave-dzhw krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506041, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:21.728829 16811 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:47:21.729765 16811 webserver.cc:460] Webserver started at http://127.0.98.62:42739/ using document root <none> and password file <none>
I20260927 10:47:21.730332 16811 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:21.730424 16811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:21.734186 16811 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:47:21.736337 16826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:21.737534 16811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:47:21.737695 16811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "52f42c39e2e74738bed67e4222acad73"
format_stamp: "Formatted at 2026-09-27 10:46:39 on dist-test-slave-dzhw"
server_key: "2cd0c87e967318a860efbe877a7ce38b"
server_key_iv: "b77921142fad963c2fd504e7a49e8a9c"
server_key_version: "encryptionkey@0"
I20260927 10:47:21.738261 16811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:21.756114 16811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:21.759188 16811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:21.759407 16811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:21.766777 16811 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37323
I20260927 10:47:21.767102 16878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37323 every 8 connection(s)
I20260927 10:47:21.767881 16811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:47:21.772104 16879 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:47:21.773693 16879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73: Bootstrap starting.
I20260927 10:47:21.775219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 16811
I20260927 10:47:21.775712 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42477
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37323
--webserver_port=36605
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:47:21.791779 16879 log.cc:824] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:21.802778 16879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:47:21.803174 16879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73: Bootstrap complete.
I20260927 10:47:21.807210 16879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f42c39e2e74738bed67e4222acad73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37323 } }
I20260927 10:47:21.807582 16879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f42c39e2e74738bed67e4222acad73, State: Initialized, Role: FOLLOWER
I20260927 10:47:21.808095 16879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f42c39e2e74738bed67e4222acad73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37323 } }
I20260927 10:47:21.808220 16879 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:47:21.808326 16879 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:47:21.808439 16879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [term 2 FOLLOWER]: Advancing to term 3
I20260927 10:47:21.810333 16879 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f42c39e2e74738bed67e4222acad73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37323 } }
I20260927 10:47:21.810671 16879 leader_election.cc:302] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52f42c39e2e74738bed67e4222acad73; no voters:
I20260927 10:47:21.811002 16879 leader_election.cc:288] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 10:47:21.811116 16883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 10:47:21.811312 16883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [term 3 LEADER]: Becoming Leader. State: Replica: 52f42c39e2e74738bed67e4222acad73, State: Running, Role: LEADER
I20260927 10:47:21.811605 16883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f42c39e2e74738bed67e4222acad73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37323 } }
I20260927 10:47:21.812212 16879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:47:21.812536 16883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52f42c39e2e74738bed67e4222acad73. Latest consensus state: current_term: 3 leader_uuid: "52f42c39e2e74738bed67e4222acad73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f42c39e2e74738bed67e4222acad73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37323 } } }
I20260927 10:47:21.812651 16883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:21.813004 16887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:47:21.813035 16884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "52f42c39e2e74738bed67e4222acad73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52f42c39e2e74738bed67e4222acad73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37323 } } }
I20260927 10:47:21.813241 16884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:21.818071 16887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:47:21.818825 16887 catalog_manager.cc:1259] Loaded cluster ID: 8bd3389dea394e959f1a07f02b555330
I20260927 10:47:21.818885 16887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:47:21.821415 16887 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:47:21.822172 16887 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73: Loaded TSK: 0
I20260927 10:47:21.822254 16887 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 52f42c39e2e74738bed67e4222acad73: Loaded TSK: 1
F20260927 10:47:21.823019 16887 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7ee564b9ad google::LogMessage::Fail() at ??:0
@ 0x7f7ee564e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f7ee564b4de google::LogMessage::Flush() at ??:0
@ 0x7f7ee564ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7ee830f18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7ee8307f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f7eeffb07bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f7eeff8a99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f7eeff8b178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f7eeffccdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f7eeffeb82b std::function<>::operator()() at ??:0
@ 0x7f7eeff8ab4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f7eeff8bf3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7eeff88579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f7eeffcc640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7ee789d3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7ee789dc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f7ee789f3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7ee788e512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7ee8ed26db start_thread at ??:0
@ 0x7f7ee414961f clone at ??:0
W20260927 10:47:21.907047 16881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:21.907374 16881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:21.907421 16881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:21.911953 16881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:21.912001 16881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:21.912062 16881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:47:21.917824 16881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36605
--tserver_master_addrs=127.0.98.62:37323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:21.918949 16881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:21.920183 16881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:21.931335 16929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:21.931339 16926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:21.931335 16927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:21.931372 16881 server_base.cc:1034] running on GCE node
I20260927 10:47:21.932055 16881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:21.932771 16881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:21.933955 16881 hybrid_clock.cc:648] HybridClock initialized: now 1790506041933929 us; error 51 us; skew 500 ppm
Sep 27 10:47:21 dist-test-slave-dzhw krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506041, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:21.936945 16881 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 10:47:21.937829 16881 webserver.cc:460] Webserver started at http://127.0.98.1:36605/ using document root <none> and password file <none>
I20260927 10:47:21.938500 16881 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:21.938596 16881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:21.942951 16881 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:47:21.945317 16936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:21.946723 16881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:47:21.946853 16881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "a5663ccff76448bab7a0118f4d2d315f"
format_stamp: "Formatted at 2026-09-27 10:46:39 on dist-test-slave-dzhw"
server_key: "c23aa5f65c6d5336923cfd3a4519e335"
server_key_iv: "25130d513879fb8f7a78f07e1dc1b4f4"
server_key_version: "encryptionkey@0"
I20260927 10:47:21.947407 16881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:21.972901 16881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:21.976128 16881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:21.976339 16881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:21.976951 16881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:21.978066 16943 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 10:47:21.983459 16881 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 10:47:21.983584 16881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 10:47:21.983695 16881 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 10:47:21.985370 16881 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 10:47:21.985448 16881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 10:47:21.985766 16943 tablet_bootstrap.cc:492] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f: Bootstrap starting.
I20260927 10:47:21.996421 16881 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42477
I20260927 10:47:21.996663 17050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42477 every 8 connection(s)
I20260927 10:47:21.997519 16881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:47:22.003016 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 16881
I20260927 10:47:22.003446 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40283
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37323
--webserver_port=33287
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:47:22.007709 16943 log.cc:824] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:22.016855 16943 tablet_bootstrap.cc:492] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:47:22.017302 16943 tablet_bootstrap.cc:492] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f: Bootstrap complete.
I20260927 10:47:22.018764 16943 ts_tablet_manager.cc:1397] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.010s
I20260927 10:47:22.022040 16943 raft_consensus.cc:348] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:47:22.022645 16943 raft_consensus.cc:729] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5663ccff76448bab7a0118f4d2d315f, State: Initialized, Role: FOLLOWER
I20260927 10:47:22.023181 16943 consensus_queue.cc:260] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:47:22.024217 16943 ts_tablet_manager.cc:1428] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260927 10:47:22.070225 17053 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:47:21.999607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.1:43743)
0927 10:47:22.000109 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:22.000164 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:22.000989 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:22.069295 (+ 68306us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":142,"threads_started":1}
W20260927 10:47:22.070909 17051 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37323 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37323: Client connection negotiation failed: client connection to 127.0.98.62:37323: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 10:47:22.071686 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:22.071391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.1:58023)
0927 10:47:22.071543 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:22.071621 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 10:47:22.072395 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:22.072185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.1:59665)
0927 10:47:22.072293 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:22.072341 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 10:47:22.072588 17051 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:47:22.118535 17054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:22.118806 17054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:22.118850 17054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:22.122891 17054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:22.122952 17054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:22.123009 17054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:47:22.128696 17054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33287
--tserver_master_addrs=127.0.98.62:37323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:22.129722 17054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:22.130777 17054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:22.140816 17062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:22.140916 17061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:22.140825 17064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:22.141664 17054 server_base.cc:1034] running on GCE node
I20260927 10:47:22.142269 17054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:22.142971 17054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:22.144229 17054 hybrid_clock.cc:648] HybridClock initialized: now 1790506042144137 us; error 94 us; skew 500 ppm
Sep 27 10:47:22 dist-test-slave-dzhw krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:22.148633 17054 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 10:47:22.149853 17054 webserver.cc:460] Webserver started at http://127.0.98.2:33287/ using document root <none> and password file <none>
I20260927 10:47:22.150764 17054 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:22.150831 17054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:22.156440 17054 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:47:22.159214 17071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:22.160709 17054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:47:22.160847 17054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "e22505803753467296b25b5d6ea2f927"
format_stamp: "Formatted at 2026-09-27 10:46:40 on dist-test-slave-dzhw"
server_key: "fe145c6006e916b10346244a0eddab0b"
server_key_iv: "cb08fb4a940d25a44887b8fc67054933"
server_key_version: "encryptionkey@0"
I20260927 10:47:22.161659 17054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:22.187615 17054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:22.197324 17054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:22.197487 17054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:22.198024 17054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:22.199290 17078 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 10:47:22.204314 17054 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 10:47:22.204402 17054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 10:47:22.204506 17054 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 10:47:22.206176 17054 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 10:47:22.206249 17054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 10:47:22.206579 17078 tablet_bootstrap.cc:492] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927: Bootstrap starting.
I20260927 10:47:22.211988 17078 log.cc:824] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:22.216499 17054 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40283
I20260927 10:47:22.216564 17186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40283 every 8 connection(s)
I20260927 10:47:22.217344 17054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:47:22.220144 17078 tablet_bootstrap.cc:492] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:47:22.220580 17078 tablet_bootstrap.cc:492] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927: Bootstrap complete.
I20260927 10:47:22.220606 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:22.219284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.2:34807)
0927 10:47:22.219803 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:22.219903 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":159,"threads_started":1}
I20260927 10:47:22.220995 17078 ts_tablet_manager.cc:1397] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
W20260927 10:47:22.221125 17187 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37323 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37323: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
I20260927 10:47:22.221165 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 17054
I20260927 10:47:22.221637 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36377
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37323
--webserver_port=37773
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:47:22.221774 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:22.221526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.2:41611)
0927 10:47:22.221652 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:22.221708 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 10:47:22.222577 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:22.222280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.2:45361)
0927 10:47:22.222435 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:22.222499 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 10:47:22.222824 17187 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 10:47:22.224298 17078 raft_consensus.cc:348] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:47:22.224741 17078 raft_consensus.cc:729] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e22505803753467296b25b5d6ea2f927, State: Initialized, Role: FOLLOWER
I20260927 10:47:22.225214 17078 consensus_queue.cc:260] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:47:22.226287 17078 ts_tablet_manager.cc:1428] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260927 10:47:22.226562 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:22.226218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.2:46111)
0927 10:47:22.226368 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:22.226471 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 10:47:22.227286 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:22.226993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.2:48065)
0927 10:47:22.227146 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:22.227209 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 10:47:22.336026 17190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:22.336396 17190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:22.336463 17190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:22.340843 17190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:22.340910 17190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:22.340973 17190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:47:22.346282 17190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37773
--tserver_master_addrs=127.0.98.62:37323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:22.347355 17190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:22.348374 17190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:22.358129 17197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:22.358101 17196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:22.358271 17199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:22.358371 17190 server_base.cc:1034] running on GCE node
I20260927 10:47:22.358834 17190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:22.359464 17190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:22.360620 17190 hybrid_clock.cc:648] HybridClock initialized: now 1790506042360607 us; error 26 us; skew 500 ppm
Sep 27 10:47:22 dist-test-slave-dzhw krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:22.363301 17190 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 10:47:22.364159 17190 webserver.cc:460] Webserver started at http://127.0.98.3:37773/ using document root <none> and password file <none>
I20260927 10:47:22.364660 17190 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:22.364718 17190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:22.368513 17190 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:47:22.370533 17206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:22.371724 17190 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:47:22.371856 17190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "fc6fe33c2104438eb903f1ec0d89f221"
format_stamp: "Formatted at 2026-09-27 10:46:40 on dist-test-slave-dzhw"
server_key: "c291b54b396fe0e92a0e1fab372a57a7"
server_key_iv: "dc55ad3b0ed46668b4300a51129b235b"
server_key_version: "encryptionkey@0"
I20260927 10:47:22.372390 17190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:22.387845 17190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:22.390929 17190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:22.391158 17190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:22.391776 17190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:22.392629 17190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:22.392701 17190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:22.392784 17190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:22.392830 17190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:22.402077 17190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36377
I20260927 10:47:22.402103 17319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36377 every 8 connection(s)
I20260927 10:47:22.403141 17190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:47:22.406742 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:22.405170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.3:47987)
0927 10:47:22.405825 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:22.405930 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":111,"threads_started":1}
W20260927 10:47:22.407208 17320 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37323 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37323: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
I20260927 10:47:22.407814 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:22.407615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.3:38423)
0927 10:47:22.407720 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:22.407764 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 10:47:22.408401 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:22.408225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.3:34093)
0927 10:47:22.408317 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:22.408357 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 10:47:22.408649 17320 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 10:47:22.409128 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 17190
I20260927 10:47:23.073705 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.073222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.1:35895)
0927 10:47:23.073508 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.073591 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":81,"threads_started":1}
I20260927 10:47:23.225073 17325 raft_consensus.cc:482] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:47:23.225243 17325 raft_consensus.cc:504] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:47:23.226289 17325 leader_election.cc:288] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fc6fe33c2104438eb903f1ec0d89f221 (127.0.98.3:36377), e22505803753467296b25b5d6ea2f927 (127.0.98.2:40283)
I20260927 10:47:23.228256 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.227869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.2:58263)
0927 10:47:23.228095 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.228156 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":48,"threads_started":1}
Sep 27 10:47:23 dist-test-slave-dzhw krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506041, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 27 10:47:23 dist-test-slave-dzhw krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506041, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 10:47:23.240315 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.226723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51587 (local address 127.0.98.3:36377)
0927 10:47:23.227102 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:23.227116 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:23.227160 (+ 44us) server_negotiation.cc:407] Connection header received
0927 10:47:23.227999 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:23.228006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:23.228362 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:23.228806 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:23.229440 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.230373 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.231672 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.232251 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.232752 (+ 501us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.235598 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:23.235603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:23.235613 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:23.235641 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:23.237489 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.238028 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.238033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.238036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.238147 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.238610 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.238616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.238618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.239039 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:23.239181 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:23.239800 (+ 619us) server_negotiation.cc:299] Negotiation successful
0927 10:47:23.240083 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":98,"threads_started":1}
I20260927 10:47:23.240302 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.226663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40283 (local address 127.0.98.1:46805)
0927 10:47:23.227012 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.227026 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:23.227128 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:23.228288 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:23.228295 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:23.228793 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:23.229258 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:23.229273 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.230590 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.230593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:23.231540 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:23.231550 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.232415 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.232417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:23.232610 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.233374 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:23.233390 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:23.235041 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:23.237429 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:23.237439 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:23.237452 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:23.237872 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:23.238287 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:23.238289 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:23.238290 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:23.238477 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:23.239162 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:23.239169 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:23.239440 (+ 271us) client_negotiation.cc:769] Sending connection context
0927 10:47:23.239633 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 10:47:23.240048 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":197,"spinlock_wait_cycles":20480,"thread_start_us":106,"threads_started":1}
I20260927 10:47:23.240324 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.226841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46805 (local address 127.0.98.2:40283)
0927 10:47:23.227142 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:23.227152 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:23.227180 (+ 28us) server_negotiation.cc:407] Connection header received
0927 10:47:23.227434 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:23.227440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:23.227868 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:23.228189 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:23.229423 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.230365 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.231695 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.232259 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.232439 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.235273 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:23.235280 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:23.235298 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:23.235338 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:23.237293 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.238005 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.238013 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.238036 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.238154 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.238605 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.238610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.238612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.239041 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:23.239182 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:23.239825 (+ 643us) server_negotiation.cc:299] Negotiation successful
0927 10:47:23.240110 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":90,"threads_started":1}
I20260927 10:47:23.240310 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.226512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36377 (local address 127.0.98.1:51587)
0927 10:47:23.226746 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.226763 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:23.226873 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:23.228606 (+ 1733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:23.228609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:23.228857 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:23.229259 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:23.229273 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.230578 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.230583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:23.231542 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:23.231551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.232393 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.232398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:23.232607 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.233423 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:23.233435 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:23.235389 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:23.237616 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:23.237619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:23.237620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:23.237880 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:23.238250 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:23.238254 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:23.238256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:23.238467 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:23.239166 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:23.239169 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:23.239441 (+ 272us) client_negotiation.cc:769] Sending connection context
0927 10:47:23.239691 (+ 250us) client_negotiation.cc:240] Negotiation successful
0927 10:47:23.240141 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":313,"spinlock_wait_cycles":20352}
I20260927 10:47:23.241732 17140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6670e2917f674a4c83f16b4105e332d8" candidate_uuid: "a5663ccff76448bab7a0118f4d2d315f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e22505803753467296b25b5d6ea2f927" is_pre_election: true
I20260927 10:47:23.242075 17140 raft_consensus.cc:2454] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5663ccff76448bab7a0118f4d2d315f in term 1.
I20260927 10:47:23.243130 16937 leader_election.cc:302] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5663ccff76448bab7a0118f4d2d315f, e22505803753467296b25b5d6ea2f927; no voters:
W20260927 10:47:23.243366 16938 leader_election.cc:341] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fc6fe33c2104438eb903f1ec0d89f221 (127.0.98.3:36377): Not found: Tablet not found: 6670e2917f674a4c83f16b4105e332d8
I20260927 10:47:23.243433 17325 raft_consensus.cc:2781] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 10:47:23.243498 17325 raft_consensus.cc:482] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:47:23.243559 17325 raft_consensus.cc:3037] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:47:23.245635 17325 raft_consensus.cc:504] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:47:23.245998 17325 leader_election.cc:288] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [CANDIDATE]: Term 2 election: Requested vote from peers fc6fe33c2104438eb903f1ec0d89f221 (127.0.98.3:36377), e22505803753467296b25b5d6ea2f927 (127.0.98.2:40283)
I20260927 10:47:23.246399 17140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6670e2917f674a4c83f16b4105e332d8" candidate_uuid: "a5663ccff76448bab7a0118f4d2d315f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e22505803753467296b25b5d6ea2f927"
I20260927 10:47:23.246601 17140 raft_consensus.cc:3037] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927 [term 1 FOLLOWER]: Advancing to term 2
W20260927 10:47:23.246731 16938 leader_election.cc:341] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer fc6fe33c2104438eb903f1ec0d89f221 (127.0.98.3:36377): Not found: Tablet not found: 6670e2917f674a4c83f16b4105e332d8
I20260927 10:47:23.248626 17140 raft_consensus.cc:2454] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5663ccff76448bab7a0118f4d2d315f in term 2.
I20260927 10:47:23.248988 16937 leader_election.cc:302] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5663ccff76448bab7a0118f4d2d315f, e22505803753467296b25b5d6ea2f927; no voters: fc6fe33c2104438eb903f1ec0d89f221
I20260927 10:47:23.249191 17325 raft_consensus.cc:2781] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [term 2 FOLLOWER]: Leader election won for term 2
I20260927 10:47:23.249405 17325 raft_consensus.cc:686] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [term 2 LEADER]: Becoming Leader. State: Replica: a5663ccff76448bab7a0118f4d2d315f, State: Running, Role: LEADER
I20260927 10:47:23.249743 17325 consensus_queue.cc:237] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:47:23.250104 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.249864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.1:35253)
0927 10:47:23.249983 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.250035 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 10:47:23.409683 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.409245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.3:39181)
0927 10:47:23.409504 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.409581 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":83,"threads_started":1}
I20260927 10:47:23.823048 17140 raft_consensus.cc:1261] T 6670e2917f674a4c83f16b4105e332d8 P e22505803753467296b25b5d6ea2f927 [term 2 FOLLOWER]: Refusing update from remote peer a5663ccff76448bab7a0118f4d2d315f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260927 10:47:23.823779 17333 consensus_queue.cc:1035] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 10:47:23.825862 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.825388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.2:56917)
0927 10:47:23.825725 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.825781 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":50,"threads_started":1}
W20260927 10:47:23.828267 16938 consensus_peers.cc:487] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f -> Peer fc6fe33c2104438eb903f1ec0d89f221 (127.0.98.3:36377): Couldn't send request to peer fc6fe33c2104438eb903f1ec0d89f221. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6670e2917f674a4c83f16b4105e332d8. This is attempt 1: this message will repeat every 5th retry.
I20260927 10:47:23.828378 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.828115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.2:47189)
0927 10:47:23.828247 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.828293 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 10:47:23.828478 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.827918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.1:46901)
0927 10:47:23.828193 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.828253 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":90,"threads_started":1}
I20260927 10:47:24.270175 17342 ts_tablet_manager.cc:927] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221: Initiating tablet copy from peer a5663ccff76448bab7a0118f4d2d315f (127.0.98.1:42477)
I20260927 10:47:24.270797 17342 tablet_copy_client.cc:316] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.1:42477
Sep 27 10:47:24 dist-test-slave-dzhw krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 10:47:24.280877 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:24.271318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42477 (local address 127.0.98.3:40427)
0927 10:47:24.271630 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:24.271655 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:24.271808 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:24.272302 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:24.272307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:24.272687 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:24.272807 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:24.272813 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:24.273814 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:24.273817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:24.274515 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:24.274524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:24.275113 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:24.275116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:24.275203 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:24.275766 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:24.275778 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:24.277233 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:24.279408 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:24.279416 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:24.279421 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:24.279635 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:24.279895 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:24.279897 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:24.279898 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:24.279968 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:24.280385 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:24.280390 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:24.280490 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 10:47:24.280601 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 10:47:24.280745 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":89,"threads_started":1}
I20260927 10:47:24.280922 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:24.271516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40427 (local address 127.0.98.1:42477)
0927 10:47:24.271866 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:24.271877 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:24.271901 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:47:24.271957 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:24.271962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:24.272160 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:24.272286 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:24.272942 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:24.273690 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:24.274643 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:24.274990 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:24.275089 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:24.277366 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:24.277376 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:24.277387 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:24.277411 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:24.279317 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:24.279744 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:24.279748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:24.279749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:24.279820 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:24.280038 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:24.280040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:24.280041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:24.280292 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:24.280393 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:24.280681 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 10:47:24.280800 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":119,"threads_started":1}
I20260927 10:47:24.281975 17025 tablet_copy_service.cc:140] P a5663ccff76448bab7a0118f4d2d315f: Received BeginTabletCopySession request for tablet 6670e2917f674a4c83f16b4105e332d8 from peer fc6fe33c2104438eb903f1ec0d89f221 ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40427)
I20260927 10:47:24.282158 17025 tablet_copy_service.cc:161] P a5663ccff76448bab7a0118f4d2d315f: Beginning new tablet copy session on tablet 6670e2917f674a4c83f16b4105e332d8 from peer fc6fe33c2104438eb903f1ec0d89f221 at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40427: session id = fc6fe33c2104438eb903f1ec0d89f221-6670e2917f674a4c83f16b4105e332d8
I20260927 10:47:24.283329 17025 tablet_copy_source_session.cc:205] T 6670e2917f674a4c83f16b4105e332d8 P a5663ccff76448bab7a0118f4d2d315f: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 10:47:24.287127 17342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6670e2917f674a4c83f16b4105e332d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:24.294588 17342 tablet_copy_client.cc:791] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221: tablet copy: Starting download of 0 data blocks...
I20260927 10:47:24.294813 17342 tablet_copy_client.cc:655] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221: tablet copy: Starting download of 1 WAL segments...
I20260927 10:47:24.296612 17342 tablet_copy_client.cc:531] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 10:47:24.298305 17342 tablet_bootstrap.cc:492] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221: Bootstrap starting.
I20260927 10:47:24.303809 17342 log.cc:824] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:24.310158 17342 tablet_bootstrap.cc:492] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:47:24.310559 17342 tablet_bootstrap.cc:492] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221: Bootstrap complete.
I20260927 10:47:24.310875 17342 ts_tablet_manager.cc:1397] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.003s
I20260927 10:47:24.313289 17342 raft_consensus.cc:348] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:47:24.313553 17342 raft_consensus.cc:729] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc6fe33c2104438eb903f1ec0d89f221, State: Initialized, Role: FOLLOWER
I20260927 10:47:24.313978 17342 consensus_queue.cc:260] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc6fe33c2104438eb903f1ec0d89f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36377 } } peers { permanent_uuid: "e22505803753467296b25b5d6ea2f927" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "a5663ccff76448bab7a0118f4d2d315f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42477 } }
I20260927 10:47:24.314961 17342 ts_tablet_manager.cc:1428] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 10:47:24.315229 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:24.314930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.3:42889)
0927 10:47:24.315072 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:24.315129 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 10:47:24.315687 17025 tablet_copy_service.cc:342] P a5663ccff76448bab7a0118f4d2d315f: Request end of tablet copy session fc6fe33c2104438eb903f1ec0d89f221-6670e2917f674a4c83f16b4105e332d8 received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40427
I20260927 10:47:24.315812 17025 tablet_copy_service.cc:433] P a5663ccff76448bab7a0118f4d2d315f: ending tablet copy session fc6fe33c2104438eb903f1ec0d89f221-6670e2917f674a4c83f16b4105e332d8 on tablet 6670e2917f674a4c83f16b4105e332d8 with peer fc6fe33c2104438eb903f1ec0d89f221
I20260927 10:47:24.316002 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:24.315740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.3:54057)
0927 10:47:24.315900 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:24.315948 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 10:47:24.733564 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:24.733215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.1:44799)
0927 10:47:24.733422 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:24.733467 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":45,"threads_started":1}
I20260927 10:47:24.733594 17274 raft_consensus.cc:1203] T 6670e2917f674a4c83f16b4105e332d8 P fc6fe33c2104438eb903f1ec0d89f221 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260927 10:47:24.734572 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:24.734291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.3:56539)
0927 10:47:24.734446 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:24.734513 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 10:47:24.829469 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:24.829048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.2:44761)
0927 10:47:24.829326 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:24.829376 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":66,"threads_started":1}
I20260927 10:47:25.734777 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:25.734282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.1:51039)
0927 10:47:25.734617 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:25.734679 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":87,"threads_started":1}
I20260927 10:47:25.735919 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:25.735251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.3:43441)
0927 10:47:25.735559 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:25.735649 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":64,"threads_started":1}
I20260927 10:47:25.830946 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:25.830353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.2:55085)
0927 10:47:25.830775 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:25.830849 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":88,"threads_started":1}
I20260927 10:47:26.736250 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:26.735845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.1:33709)
0927 10:47:26.736098 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:26.736156 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":76,"threads_started":1}
I20260927 10:47:26.737161 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:26.736762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.3:40791)
0927 10:47:26.737030 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:26.737083 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":59,"threads_started":1}
I20260927 10:47:26.832288 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:26.831847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.2:48391)
0927 10:47:26.832127 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:26.832190 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
I20260927 10:47:27.737665 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:27.737227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.1:48295)
0927 10:47:27.737512 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:27.737571 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":76,"threads_started":1}
I20260927 10:47:27.738476 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:27.738224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.1:49465)
0927 10:47:27.738328 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:27.738369 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 10:47:27.738494 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:27.738066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.3:35963)
0927 10:47:27.738293 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:27.738359 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":57,"threads_started":1}
I20260927 10:47:27.833706 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:27.833261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.2:47623)
0927 10:47:27.833550 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:27.833610 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":78,"threads_started":1}
I20260927 10:47:28.739675 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:28.739150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.1:50637)
0927 10:47:28.739434 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:28.739496 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":69,"threads_started":1}
I20260927 10:47:28.739727 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:28.739220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.3:41183)
0927 10:47:28.739521 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:28.739615 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":77,"threads_started":1}
I20260927 10:47:28.835201 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:28.834744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.2:57685)
0927 10:47:28.835038 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:28.835100 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":105,"threads_started":1}
I20260927 10:47:29.740875 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:29.740471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.3:49379)
0927 10:47:29.740732 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:29.740785 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":108,"threads_started":1}
I20260927 10:47:29.740875 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:29.740476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.1:52855)
0927 10:47:29.740723 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:29.740785 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":93,"threads_started":1}
I20260927 10:47:29.836447 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:29.835991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.2:57077)
0927 10:47:29.836294 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:29.836350 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":98,"threads_started":1}
I20260927 10:47:30.742238 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:30.741722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.3:56163)
0927 10:47:30.742042 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:30.742113 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":90,"threads_started":1}
I20260927 10:47:30.742245 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:30.741714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.1:56487)
0927 10:47:30.742054 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:30.742106 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":89,"threads_started":1}
I20260927 10:47:30.837760 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:30.837293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.2:52315)
0927 10:47:30.837582 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:30.837655 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":83,"threads_started":1}
I20260927 10:47:31.743355 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:31.742957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.3:37639)
0927 10:47:31.743206 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:31.743259 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":85,"threads_started":1}
I20260927 10:47:31.743417 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:31.742961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.1:55729)
0927 10:47:31.743286 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:31.743334 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":89,"threads_started":1}
I20260927 10:47:31.839277 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:31.838739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.2:51453)
0927 10:47:31.839016 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:31.839137 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
I20260927 10:47:32.744683 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:32.744274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.1:52605)
0927 10:47:32.744536 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:32.744587 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":102,"threads_started":1}
I20260927 10:47:32.744715 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:32.744275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.3:55883)
0927 10:47:32.744583 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:32.744630 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":104,"threads_started":1}
I20260927 10:47:32.840723 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:32.840302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.2:58775)
0927 10:47:32.840559 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:32.840624 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":80,"threads_started":1}
I20260927 10:47:33.745975 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:33.745529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.3:43255)
0927 10:47:33.745786 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:33.745854 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":95,"threads_started":1}
I20260927 10:47:33.746060 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:33.745622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.1:39603)
0927 10:47:33.745886 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:33.745936 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":67,"threads_started":1}
I20260927 10:47:33.842021 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:33.841596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.2:49259)
0927 10:47:33.841865 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:33.841926 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":74,"threads_started":1}
I20260927 10:47:34.747260 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:34.746831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.1:44955)
0927 10:47:34.747110 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:34.747166 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":83,"threads_started":1}
I20260927 10:47:34.747275 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:34.746836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.3:41661)
0927 10:47:34.747140 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:34.747190 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":113,"threads_started":1}
I20260927 10:47:34.843369 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:34.842922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.2:40029)
0927 10:47:34.843198 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:34.843254 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":83,"threads_started":1}
I20260927 10:47:35.748560 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:35.748124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.3:52873)
0927 10:47:35.748399 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:35.748460 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
I20260927 10:47:35.748579 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:35.748122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.1:46901)
0927 10:47:35.748423 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:35.748495 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":108,"threads_started":1}
I20260927 10:47:35.844669 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:35.844201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.2:50473)
0927 10:47:35.844486 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:35.844554 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
I20260927 10:47:36.749989 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:36.749554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.1:60355)
0927 10:47:36.749859 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:36.749907 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":72,"threads_started":1}
I20260927 10:47:36.749977 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:36.749401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.3:51637)
0927 10:47:36.749795 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:36.749859 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":105,"threads_started":1}
I20260927 10:47:36.845961 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:36.845563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37323 (local address 127.0.98.2:38567)
0927 10:47:36.845810 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:36.845866 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":75,"threads_started":1}
I20260927 10:47:37.418874 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 16881
I20260927 10:47:37.424811 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 17054
I20260927 10:47:37.430071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 17190
2026-09-27T10:47:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59113 ms)
[----------] 30 tests from SecurityITest (245288 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:47:37 dist-test-slave-dzhw krb5kdc[17391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:47:37 dist-test-slave-dzhw krb5kdc[17391](info): set up 2 sockets
Sep 27 10:47:37 dist-test-slave-dzhw krb5kdc[17391](info): commencing operation
krb5kdc: starting...
W20260927 10:47:38.482288 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:47:38 dist-test-slave-dzhw krb5kdc[17391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:47:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:47:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:47:38.605638 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45721
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:38.715404 17404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:38.715718 17404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:38.715797 17404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:38.719774 17404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:38.719861 17404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:38.719897 17404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:38.719950 17404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:38.719986 17404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:38.725212 17404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:38.726297 17404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:38.727339 17404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:38.735890 17409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:38.736001 17410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:38.735975 17404 server_base.cc:1034] running on GCE node
W20260927 10:47:38.735914 17412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:38.736989 17404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:38.738080 17404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:38.739267 17404 hybrid_clock.cc:648] HybridClock initialized: now 1790506058739248 us; error 38 us; skew 500 ppm
Sep 27 10:47:38 dist-test-slave-dzhw krb5kdc[17391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:38.742138 17404 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:47:38.743094 17404 webserver.cc:460] Webserver started at http://127.0.98.62:34607/ using document root <none> and password file <none>
I20260927 10:47:38.743625 17404 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:38.743683 17404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:38.743872 17404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:38.745841 17404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "f3675aa08ab04b669b604e9190f8a767"
format_stamp: "Formatted at 2026-09-27 10:47:38 on dist-test-slave-dzhw"
server_key: "7ea90b6dcc2a6409428e7644fd0b328d"
server_key_iv: "09ed5cbbb75f3ff3439a58c4ea23f8a4"
server_key_version: "encryptionkey@0"
I20260927 10:47:38.746356 17404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "f3675aa08ab04b669b604e9190f8a767"
format_stamp: "Formatted at 2026-09-27 10:47:38 on dist-test-slave-dzhw"
server_key: "7ea90b6dcc2a6409428e7644fd0b328d"
server_key_iv: "09ed5cbbb75f3ff3439a58c4ea23f8a4"
server_key_version: "encryptionkey@0"
I20260927 10:47:38.749960 17404 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:47:38.752363 17419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:38.753608 17404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:47:38.753746 17404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "f3675aa08ab04b669b604e9190f8a767"
format_stamp: "Formatted at 2026-09-27 10:47:38 on dist-test-slave-dzhw"
server_key: "7ea90b6dcc2a6409428e7644fd0b328d"
server_key_iv: "09ed5cbbb75f3ff3439a58c4ea23f8a4"
server_key_version: "encryptionkey@0"
I20260927 10:47:38.753855 17404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:38.773871 17404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:38.776924 17404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:38.777125 17404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:38.784276 17404 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45721
I20260927 10:47:38.784293 17471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45721 every 8 connection(s)
I20260927 10:47:38.785295 17404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:47:38.788051 17472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:38.792248 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 17404
I20260927 10:47:38.792349 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:47:38.792662 392 external_mini_cluster.cc:1428] Setting key 54832147e6004e2368a45c6ed72118a7
I20260927 10:47:38.793692 17472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3675aa08ab04b669b604e9190f8a767: Bootstrap starting.
I20260927 10:47:38.796001 17472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3675aa08ab04b669b604e9190f8a767: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:38.796715 17472 log.cc:824] T 00000000000000000000000000000000 P f3675aa08ab04b669b604e9190f8a767: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:38.798580 17472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3675aa08ab04b669b604e9190f8a767: No bootstrap required, opened a new log
Sep 27 10:47:38 dist-test-slave-dzhw krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:47:38.802476 17472 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3675aa08ab04b669b604e9190f8a767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3675aa08ab04b669b604e9190f8a767" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45721 } }
I20260927 10:47:38.802713 17472 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3675aa08ab04b669b604e9190f8a767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:38.802793 17472 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3675aa08ab04b669b604e9190f8a767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3675aa08ab04b669b604e9190f8a767, State: Initialized, Role: FOLLOWER
I20260927 10:47:38.803262 17472 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3675aa08ab04b669b604e9190f8a767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3675aa08ab04b669b604e9190f8a767" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45721 } }
I20260927 10:47:38.803386 17472 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3675aa08ab04b669b604e9190f8a767 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:47:38.803469 17472 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3675aa08ab04b669b604e9190f8a767 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:47:38.803588 17472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3675aa08ab04b669b604e9190f8a767 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:38.804399 17472 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3675aa08ab04b669b604e9190f8a767 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3675aa08ab04b669b604e9190f8a767" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45721 } }
I20260927 10:47:38.804713 17472 leader_election.cc:302] T 00000000000000000000000000000000 P f3675aa08ab04b669b604e9190f8a767 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3675aa08ab04b669b604e9190f8a767; no voters:
I20260927 10:47:38.805003 17472 leader_election.cc:288] T 00000000000000000000000000000000 P f3675aa08ab04b669b604e9190f8a767 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:47:38.805105 17477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3675aa08ab04b669b604e9190f8a767 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:38.805305 17477 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3675aa08ab04b669b604e9190f8a767 [term 1 LEADER]: Becoming Leader. State: Replica: f3675aa08ab04b669b604e9190f8a767, State: Running, Role: LEADER
I20260927 10:47:38.805717 17477 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3675aa08ab04b669b604e9190f8a767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3675aa08ab04b669b604e9190f8a767" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45721 } }
I20260927 10:47:38.806283 17472 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3675aa08ab04b669b604e9190f8a767 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:47:38.806941 17477 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3675aa08ab04b669b604e9190f8a767 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3675aa08ab04b669b604e9190f8a767. Latest consensus state: current_term: 1 leader_uuid: "f3675aa08ab04b669b604e9190f8a767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3675aa08ab04b669b604e9190f8a767" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45721 } } }
I20260927 10:47:38.807659 17477 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3675aa08ab04b669b604e9190f8a767 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:38.807721 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:38.794056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59968 (local address 127.0.98.62:45721)
0927 10:47:38.794516 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:38.794532 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:38.794565 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:47:38.795190 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:38.795207 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:38.795577 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:38.795909 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:38.796684 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:38.797691 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:38.798600 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:38.799115 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:38.799289 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:38.801962 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:38.801967 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:38.801980 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:38.802005 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:38.803939 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:38.804351 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:38.804357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:38.804363 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:38.804520 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:38.804772 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:38.804776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:38.804777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:38.805305 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:38.806206 (+ 901us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:38.806348 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 10:47:38.806697 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":115,"threads_started":1}
I20260927 10:47:38.807536 17478 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3675aa08ab04b669b604e9190f8a767 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3675aa08ab04b669b604e9190f8a767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3675aa08ab04b669b604e9190f8a767" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45721 } } }
I20260927 10:47:38.808835 17478 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3675aa08ab04b669b604e9190f8a767 [sys.catalog]: This master's current role is: LEADER
W20260927 10:47:38.811312 17492 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f3675aa08ab04b669b604e9190f8a767: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:47:38.811390 17492 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:47:38.811465 17491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:47:38.812165 17491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:47:38.817883 17491 catalog_manager.cc:1347] Generated new cluster ID: 158e8cad38b641c2b8dd1d807076bf5e
I20260927 10:47:38.817965 17491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:47:38.847470 17491 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:47:38.848477 17491 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:47:38.852202 17491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3675aa08ab04b669b604e9190f8a767: Generated new TSK 0
I20260927 10:47:38.852836 17491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 10:47:38.886227 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45721
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:39.012565 17498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:39.012810 17498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:39.012853 17498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:39.016775 17498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:39.016835 17498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:39.016892 17498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:47:39.022039 17498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:39.023118 17498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:39.024065 17498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:39.033368 17506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:39.033368 17503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:39.033407 17504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:39.033753 17498 server_base.cc:1034] running on GCE node
I20260927 10:47:39.034159 17498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:39.034842 17498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:39.036062 17498 hybrid_clock.cc:648] HybridClock initialized: now 1790506059036046 us; error 33 us; skew 500 ppm
Sep 27 10:47:39 dist-test-slave-dzhw krb5kdc[17391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:39.038938 17498 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 10:47:39.039749 17498 webserver.cc:460] Webserver started at http://127.0.98.1:43985/ using document root <none> and password file <none>
I20260927 10:47:39.040282 17498 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:39.040338 17498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:39.040504 17498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:39.042505 17498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "7fac15de23fb47ddaf2f70f1665a825e"
format_stamp: "Formatted at 2026-09-27 10:47:39 on dist-test-slave-dzhw"
server_key: "ad4cff7cadead6733fdb601810d63c0c"
server_key_iv: "b1cf59178a11ae67d76b39b6bd9f01db"
server_key_version: "encryptionkey@0"
I20260927 10:47:39.043021 17498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7fac15de23fb47ddaf2f70f1665a825e"
format_stamp: "Formatted at 2026-09-27 10:47:39 on dist-test-slave-dzhw"
server_key: "ad4cff7cadead6733fdb601810d63c0c"
server_key_iv: "b1cf59178a11ae67d76b39b6bd9f01db"
server_key_version: "encryptionkey@0"
I20260927 10:47:39.046612 17498 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:47:39.048969 17513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:39.050144 17498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:47:39.050235 17498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "7fac15de23fb47ddaf2f70f1665a825e"
format_stamp: "Formatted at 2026-09-27 10:47:39 on dist-test-slave-dzhw"
server_key: "ad4cff7cadead6733fdb601810d63c0c"
server_key_iv: "b1cf59178a11ae67d76b39b6bd9f01db"
server_key_version: "encryptionkey@0"
I20260927 10:47:39.050313 17498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:39.061650 17498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:39.064733 17498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:39.064875 17498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:39.065366 17498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:39.066092 17498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:39.066138 17498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:39.066177 17498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:39.066192 17498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:39.074796 17498 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42373
I20260927 10:47:39.074819 17626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42373 every 8 connection(s)
I20260927 10:47:39.075629 17498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:47:39.083500 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 17498
I20260927 10:47:39.083632 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:47:39.083904 392 external_mini_cluster.cc:1428] Setting key 8766d55687c0fc5915f14a323afc1626
Sep 27 10:47:39 dist-test-slave-dzhw krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:47:39.090161 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:39.077375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57399 (local address 127.0.98.62:45721)
0927 10:47:39.077505 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:39.077509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:39.078339 (+ 830us) server_negotiation.cc:407] Connection header received
0927 10:47:39.079643 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:39.079646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:39.079688 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:39.079760 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:39.080874 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.081406 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.082545 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.082771 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.082887 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:39.086183 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:39.086186 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:39.086189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:39.086214 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:39.088003 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:39.088609 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:39.088611 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:39.088612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:39.088683 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:39.089053 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:39.089055 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:39.089056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:39.089210 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:39.089281 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:39.089939 (+ 658us) server_negotiation.cc:299] Negotiation successful
0927 10:47:39.090042 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:47:39.091341 17629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:39.077736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45721 (local address 127.0.98.1:57399)
0927 10:47:39.078191 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:39.078233 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:39.079464 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:39.079927 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:39.079934 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:39.080264 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:39.080749 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:39.080761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.081515 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.081518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:39.082386 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:39.082395 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.082923 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.082926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:39.083061 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:39.083747 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:39.083774 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:39.086017 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:39.088147 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:39.088156 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:39.088172 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:39.088506 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:39.088768 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:39.088774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:39.088779 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:39.088964 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:39.089312 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:39.089321 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:39.089653 (+ 332us) client_negotiation.cc:769] Sending connection context
0927 10:47:39.089871 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 10:47:39.090218 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":95,"threads_started":1}
I20260927 10:47:39.092566 17627 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45721
I20260927 10:47:39.092792 17627 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:39.093254 17627 heartbeater.cc:511] Master 127.0.98.62:45721 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 10:47:39.094869 17436 ts_manager.cc:194] Registered new tserver with Master: 7fac15de23fb47ddaf2f70f1665a825e (127.0.98.1:42373)
I20260927 10:47:39.095948 17436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 10:47:39.111290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45721
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:39.218364 17632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:39.218674 17632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:39.218746 17632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:39.222898 17632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:39.222973 17632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:39.223057 17632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:47:39.228257 17632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:39.229283 17632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:39.230259 17632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:39.239959 17640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:39.239965 17637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:39.239965 17638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:39.240126 17632 server_base.cc:1034] running on GCE node
I20260927 10:47:39.240621 17632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:39.241297 17632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:39.242707 17632 hybrid_clock.cc:648] HybridClock initialized: now 1790506059242664 us; error 88 us; skew 500 ppm
Sep 27 10:47:39 dist-test-slave-dzhw krb5kdc[17391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:39.246317 17632 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 10:47:39.247236 17632 webserver.cc:460] Webserver started at http://127.0.98.2:40593/ using document root <none> and password file <none>
I20260927 10:47:39.247825 17632 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:39.247921 17632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:39.248150 17632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:39.250197 17632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "587b34324562462d96146cec8e7e8ed3"
format_stamp: "Formatted at 2026-09-27 10:47:39 on dist-test-slave-dzhw"
server_key: "9548eb9c38f2a71e5ff268531e92f5cc"
server_key_iv: "576d09b1dda4512fb55a613a9b57a371"
server_key_version: "encryptionkey@0"
I20260927 10:47:39.250823 17632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "587b34324562462d96146cec8e7e8ed3"
format_stamp: "Formatted at 2026-09-27 10:47:39 on dist-test-slave-dzhw"
server_key: "9548eb9c38f2a71e5ff268531e92f5cc"
server_key_iv: "576d09b1dda4512fb55a613a9b57a371"
server_key_version: "encryptionkey@0"
I20260927 10:47:39.254817 17632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 10:47:39.257218 17647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:39.258587 17632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:47:39.258719 17632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "587b34324562462d96146cec8e7e8ed3"
format_stamp: "Formatted at 2026-09-27 10:47:39 on dist-test-slave-dzhw"
server_key: "9548eb9c38f2a71e5ff268531e92f5cc"
server_key_iv: "576d09b1dda4512fb55a613a9b57a371"
server_key_version: "encryptionkey@0"
I20260927 10:47:39.258836 17632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:39.279547 17632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:39.282372 17632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:39.282612 17632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:39.283193 17632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:39.283996 17632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:39.284068 17632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:39.284137 17632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:39.284188 17632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:39.293056 17632 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34805
I20260927 10:47:39.293083 17760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34805 every 8 connection(s)
I20260927 10:47:39.293915 17632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:47:39.297524 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 17632
I20260927 10:47:39.297662 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:47:39.297981 392 external_mini_cluster.cc:1428] Setting key bf62c1b612d88d3475d8427934b8dfe6
Sep 27 10:47:39 dist-test-slave-dzhw krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 10:47:39.308314 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:39.295668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43603 (local address 127.0.98.62:45721)
0927 10:47:39.295814 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:39.295818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:39.296689 (+ 871us) server_negotiation.cc:407] Connection header received
0927 10:47:39.298309 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:39.298312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:39.298394 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:39.298517 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:39.299965 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.300479 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.301835 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.302068 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.302181 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:39.304610 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:39.304614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:39.304617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:39.304647 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:39.306084 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:39.306688 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:39.306693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:39.306695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:39.306774 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:39.307107 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:39.307111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:39.307113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:39.307294 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:39.307401 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:39.308016 (+ 615us) server_negotiation.cc:299] Negotiation successful
0927 10:47:39.308144 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:47:39.309299 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:39.296084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45721 (local address 127.0.98.2:43603)
0927 10:47:39.296536 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:39.296579 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:39.298066 (+ 1487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:39.298706 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:39.298717 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:39.299125 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:39.299810 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:39.299830 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.300606 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.300612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:39.301702 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:39.301711 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.302216 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.302221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:39.302386 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:39.303053 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:39.303069 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:39.304453 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:39.306216 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:39.306221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:39.306236 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:39.306586 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:39.306872 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:39.306875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:39.306877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:39.307012 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:39.307398 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:39.307415 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:39.307729 (+ 314us) client_negotiation.cc:769] Sending connection context
0927 10:47:39.307976 (+ 247us) client_negotiation.cc:240] Negotiation successful
0927 10:47:39.308290 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":104,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 10:47:39.310468 17761 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45721
I20260927 10:47:39.310698 17761 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:39.311306 17761 heartbeater.cc:511] Master 127.0.98.62:45721 requested a full tablet report, sending...
I20260927 10:47:39.312393 17436 ts_manager.cc:194] Registered new tserver with Master: 587b34324562462d96146cec8e7e8ed3 (127.0.98.2:34805)
I20260927 10:47:39.312964 17436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 10:47:39.325814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45721
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:39.436014 17766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:39.436265 17766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:39.436350 17766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:39.440371 17766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:39.440431 17766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:39.440557 17766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:47:39.445779 17766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:39.446856 17766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:39.447857 17766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:39.457536 17774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:39.457540 17771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:39.457551 17772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:39.457844 17766 server_base.cc:1034] running on GCE node
I20260927 10:47:39.458360 17766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:39.459061 17766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:39.460242 17766 hybrid_clock.cc:648] HybridClock initialized: now 1790506059460222 us; error 34 us; skew 500 ppm
Sep 27 10:47:39 dist-test-slave-dzhw krb5kdc[17391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:39.463428 17766 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 10:47:39.464308 17766 webserver.cc:460] Webserver started at http://127.0.98.3:37343/ using document root <none> and password file <none>
I20260927 10:47:39.464890 17766 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:39.464952 17766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:39.465166 17766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:39.467337 17766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "c285c2aeffcc4c1893c721207b324e89"
format_stamp: "Formatted at 2026-09-27 10:47:39 on dist-test-slave-dzhw"
server_key: "72817a4007b41d6efb09d6e6a548f83d"
server_key_iv: "0af4ce126dafe9cd0b0d6ccdcddf23e0"
server_key_version: "encryptionkey@0"
I20260927 10:47:39.467916 17766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c285c2aeffcc4c1893c721207b324e89"
format_stamp: "Formatted at 2026-09-27 10:47:39 on dist-test-slave-dzhw"
server_key: "72817a4007b41d6efb09d6e6a548f83d"
server_key_iv: "0af4ce126dafe9cd0b0d6ccdcddf23e0"
server_key_version: "encryptionkey@0"
I20260927 10:47:39.471697 17766 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:47:39.474308 17781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:39.475703 17766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:47:39.475848 17766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "c285c2aeffcc4c1893c721207b324e89"
format_stamp: "Formatted at 2026-09-27 10:47:39 on dist-test-slave-dzhw"
server_key: "72817a4007b41d6efb09d6e6a548f83d"
server_key_iv: "0af4ce126dafe9cd0b0d6ccdcddf23e0"
server_key_version: "encryptionkey@0"
I20260927 10:47:39.475979 17766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:39.504743 17766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:39.508044 17766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:39.508258 17766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:39.508862 17766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:39.509634 17766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:39.509680 17766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:39.509719 17766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:39.509734 17766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:39.518885 17766 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42363
I20260927 10:47:39.518947 17894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42363 every 8 connection(s)
I20260927 10:47:39.519790 17766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:47:39.522527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 17766
I20260927 10:47:39.522683 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:47:39.522972 392 external_mini_cluster.cc:1428] Setting key 58ab506a2d9e3744d123fccc8f62d217
Sep 27 10:47:39 dist-test-slave-dzhw krb5kdc[17391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:47:39.533856 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:39.521733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57285 (local address 127.0.98.62:45721)
0927 10:47:39.521917 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:39.521922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:39.522752 (+ 830us) server_negotiation.cc:407] Connection header received
0927 10:47:39.524312 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:39.524315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:39.524383 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:39.524473 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:39.525856 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.526388 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.527569 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.527782 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.527868 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:39.530380 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:39.530383 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:39.530385 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:39.530488 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:39.531798 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:39.532301 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:39.532306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:39.532308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:39.532396 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:39.532733 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:39.532738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:39.532740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:39.532947 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:39.533057 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:39.533559 (+ 502us) server_negotiation.cc:299] Negotiation successful
0927 10:47:39.533683 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 10:47:39.534711 17897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:39.522086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45721 (local address 127.0.98.3:57285)
0927 10:47:39.522598 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:39.522655 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:39.524075 (+ 1420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:39.524672 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:39.524680 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:39.525191 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:39.525713 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:39.525726 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.526570 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.526573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:39.527447 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:39.527455 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.527915 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.527918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:39.528049 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:39.528691 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:39.528707 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:39.530198 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:39.531926 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:39.531931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:39.531947 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:39.532194 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:39.532496 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:39.532499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:39.532501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:39.532635 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:39.533055 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:39.533062 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:39.533307 (+ 245us) client_negotiation.cc:769] Sending connection context
0927 10:47:39.533504 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 10:47:39.533820 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":145,"threads_started":1}
I20260927 10:47:39.535933 17895 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45721
I20260927 10:47:39.536187 17895 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:39.536746 17895 heartbeater.cc:511] Master 127.0.98.62:45721 requested a full tablet report, sending...
I20260927 10:47:39.537722 17436 ts_manager.cc:194] Registered new tserver with Master: c285c2aeffcc4c1893c721207b324e89 (127.0.98.3:42363)
I20260927 10:47:39.538295 17436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57285
I20260927 10:47:39.548805 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:47:39.558534 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:39.550796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59982 (local address 127.0.98.62:45721)
0927 10:47:39.550944 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:39.550949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:39.551104 (+ 155us) server_negotiation.cc:407] Connection header received
0927 10:47:39.551259 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:39.551262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:39.551304 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:39.551401 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:39.552113 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.552666 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.553463 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.553747 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.553827 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:39.555455 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:39.555458 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:39.555460 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:39.555481 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:39.556740 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:39.557186 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:39.557191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:39.557193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:39.557276 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:39.557556 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:39.557560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:39.557562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:39.557776 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:39.557915 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:39.558165 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 10:47:39.558292 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:47:39.561250 17436 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:47:39.564541 17436 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:47:39.579071 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:39.573153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49892 (local address 127.0.98.1:42373)
0927 10:47:39.573594 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:39.573608 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:39.573633 (+ 25us) server_negotiation.cc:407] Connection header received
0927 10:47:39.573852 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:39.573858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:39.574079 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:39.574261 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:39.574653 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.575901 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.577785 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.578318 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.578464 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:39.578613 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:39.578799 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 10:47:39.578899 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":88,"threads_started":1}
I20260927 10:47:39.579097 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:39.572770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42373 (local address 127.0.0.1:49892)
0927 10:47:39.573489 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:39.573531 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:39.573654 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:39.574248 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:39.574254 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:39.574276 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:39.574495 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:39.574505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.576059 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.576064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:39.577656 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:39.577665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.578471 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.578474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:39.578577 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:39.578620 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 10:47:39.578767 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 10:47:39.578935 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":342,"thread_start_us":103,"threads_started":1}
I20260927 10:47:39.579258 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:39.573410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42363 (local address 127.0.0.1:52790)
0927 10:47:39.574377 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:39.574392 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:39.574508 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:39.574973 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:39.574976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:39.574995 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:39.575076 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:39.575082 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.576404 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.576410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:39.577437 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:39.577447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.578712 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.578716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:39.578942 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:39.578958 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 10:47:39.579033 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 10:47:39.579119 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":611,"thread_start_us":87,"threads_started":1}
I20260927 10:47:39.579319 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:39.572840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37994 (local address 127.0.98.2:34805)
0927 10:47:39.573408 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:39.573420 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:39.573796 (+ 376us) server_negotiation.cc:407] Connection header received
0927 10:47:39.574023 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:39.574029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:39.574215 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:39.574344 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:39.574802 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.575952 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.577653 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.578389 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.578511 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:39.578645 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:39.579095 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 10:47:39.579179 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":106,"threads_started":1}
I20260927 10:47:39.579733 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:39.572709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34805 (local address 127.0.0.1:37994)
0927 10:47:39.573292 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:39.573717 (+ 425us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:39.573866 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:39.574583 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:39.574586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:39.574604 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:39.574693 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:39.574698 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.576086 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.576089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:39.577548 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:39.577559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.578712 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.578716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:39.578917 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:39.578957 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 10:47:39.579521 (+ 564us) client_negotiation.cc:240] Negotiation successful
0927 10:47:39.579599 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":109,"thread_start_us":74,"threads_started":1}
I20260927 10:47:39.580011 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:39.573634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52790 (local address 127.0.98.3:42363)
0927 10:47:39.574241 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:39.574252 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:39.574498 (+ 246us) server_negotiation.cc:407] Connection header received
0927 10:47:39.574653 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:39.574658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:39.574824 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:39.574980 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:39.575205 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.576270 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.577560 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.578353 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.578526 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:39.578645 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:39.579757 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0927 10:47:39.579865 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":156,"threads_started":1}
I20260927 10:47:39.582144 17561 tablet_service.cc:1467] Processing CreateTablet for tablet 029649a85680402eb91045d5ab8d4e8a (DEFAULT_TABLE table=test-table [id=14caea946b264a5bb20163f085e03653]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:39.582526 17695 tablet_service.cc:1467] Processing CreateTablet for tablet 029649a85680402eb91045d5ab8d4e8a (DEFAULT_TABLE table=test-table [id=14caea946b264a5bb20163f085e03653]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:39.582526 17829 tablet_service.cc:1467] Processing CreateTablet for tablet 029649a85680402eb91045d5ab8d4e8a (DEFAULT_TABLE table=test-table [id=14caea946b264a5bb20163f085e03653]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:39.583230 17561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 029649a85680402eb91045d5ab8d4e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:39.583479 17829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 029649a85680402eb91045d5ab8d4e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:39.583479 17695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 029649a85680402eb91045d5ab8d4e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:39.589082 17912 tablet_bootstrap.cc:492] T 029649a85680402eb91045d5ab8d4e8a P 7fac15de23fb47ddaf2f70f1665a825e: Bootstrap starting.
I20260927 10:47:39.590654 17913 tablet_bootstrap.cc:492] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89: Bootstrap starting.
I20260927 10:47:39.590981 17914 tablet_bootstrap.cc:492] T 029649a85680402eb91045d5ab8d4e8a P 587b34324562462d96146cec8e7e8ed3: Bootstrap starting.
I20260927 10:47:39.591506 17912 tablet_bootstrap.cc:654] T 029649a85680402eb91045d5ab8d4e8a P 7fac15de23fb47ddaf2f70f1665a825e: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:39.592659 17912 log.cc:824] T 029649a85680402eb91045d5ab8d4e8a P 7fac15de23fb47ddaf2f70f1665a825e: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:39.592967 17913 tablet_bootstrap.cc:654] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:39.592967 17914 tablet_bootstrap.cc:654] T 029649a85680402eb91045d5ab8d4e8a P 587b34324562462d96146cec8e7e8ed3: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:39.594241 17914 log.cc:824] T 029649a85680402eb91045d5ab8d4e8a P 587b34324562462d96146cec8e7e8ed3: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:39.594240 17913 log.cc:824] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:39.594291 17912 tablet_bootstrap.cc:492] T 029649a85680402eb91045d5ab8d4e8a P 7fac15de23fb47ddaf2f70f1665a825e: No bootstrap required, opened a new log
I20260927 10:47:39.594488 17912 ts_tablet_manager.cc:1397] T 029649a85680402eb91045d5ab8d4e8a P 7fac15de23fb47ddaf2f70f1665a825e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:47:39.596117 17913 tablet_bootstrap.cc:492] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89: No bootstrap required, opened a new log
I20260927 10:47:39.596297 17913 ts_tablet_manager.cc:1397] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:47:39.596457 17914 tablet_bootstrap.cc:492] T 029649a85680402eb91045d5ab8d4e8a P 587b34324562462d96146cec8e7e8ed3: No bootstrap required, opened a new log
I20260927 10:47:39.596627 17914 ts_tablet_manager.cc:1397] T 029649a85680402eb91045d5ab8d4e8a P 587b34324562462d96146cec8e7e8ed3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:47:39.598013 17912 raft_consensus.cc:348] T 029649a85680402eb91045d5ab8d4e8a P 7fac15de23fb47ddaf2f70f1665a825e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587b34324562462d96146cec8e7e8ed3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34805 } } peers { permanent_uuid: "7fac15de23fb47ddaf2f70f1665a825e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42373 } } peers { permanent_uuid: "c285c2aeffcc4c1893c721207b324e89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42363 } }
I20260927 10:47:39.598253 17912 raft_consensus.cc:374] T 029649a85680402eb91045d5ab8d4e8a P 7fac15de23fb47ddaf2f70f1665a825e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:39.598326 17912 raft_consensus.cc:729] T 029649a85680402eb91045d5ab8d4e8a P 7fac15de23fb47ddaf2f70f1665a825e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fac15de23fb47ddaf2f70f1665a825e, State: Initialized, Role: FOLLOWER
I20260927 10:47:39.598803 17912 consensus_queue.cc:260] T 029649a85680402eb91045d5ab8d4e8a P 7fac15de23fb47ddaf2f70f1665a825e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587b34324562462d96146cec8e7e8ed3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34805 } } peers { permanent_uuid: "7fac15de23fb47ddaf2f70f1665a825e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42373 } } peers { permanent_uuid: "c285c2aeffcc4c1893c721207b324e89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42363 } }
I20260927 10:47:39.599462 17627 heartbeater.cc:503] Master 127.0.98.62:45721 was elected leader, sending a full tablet report...
I20260927 10:47:39.599807 17912 ts_tablet_manager.cc:1428] T 029649a85680402eb91045d5ab8d4e8a P 7fac15de23fb47ddaf2f70f1665a825e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:47:39.600131 17913 raft_consensus.cc:348] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587b34324562462d96146cec8e7e8ed3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34805 } } peers { permanent_uuid: "7fac15de23fb47ddaf2f70f1665a825e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42373 } } peers { permanent_uuid: "c285c2aeffcc4c1893c721207b324e89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42363 } }
I20260927 10:47:39.600214 17914 raft_consensus.cc:348] T 029649a85680402eb91045d5ab8d4e8a P 587b34324562462d96146cec8e7e8ed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587b34324562462d96146cec8e7e8ed3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34805 } } peers { permanent_uuid: "7fac15de23fb47ddaf2f70f1665a825e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42373 } } peers { permanent_uuid: "c285c2aeffcc4c1893c721207b324e89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42363 } }
I20260927 10:47:39.600431 17914 raft_consensus.cc:374] T 029649a85680402eb91045d5ab8d4e8a P 587b34324562462d96146cec8e7e8ed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:39.600512 17913 raft_consensus.cc:374] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:39.600521 17914 raft_consensus.cc:729] T 029649a85680402eb91045d5ab8d4e8a P 587b34324562462d96146cec8e7e8ed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 587b34324562462d96146cec8e7e8ed3, State: Initialized, Role: FOLLOWER
I20260927 10:47:39.600579 17913 raft_consensus.cc:729] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c285c2aeffcc4c1893c721207b324e89, State: Initialized, Role: FOLLOWER
I20260927 10:47:39.600955 17914 consensus_queue.cc:260] T 029649a85680402eb91045d5ab8d4e8a P 587b34324562462d96146cec8e7e8ed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587b34324562462d96146cec8e7e8ed3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34805 } } peers { permanent_uuid: "7fac15de23fb47ddaf2f70f1665a825e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42373 } } peers { permanent_uuid: "c285c2aeffcc4c1893c721207b324e89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42363 } }
I20260927 10:47:39.600955 17913 consensus_queue.cc:260] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587b34324562462d96146cec8e7e8ed3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34805 } } peers { permanent_uuid: "7fac15de23fb47ddaf2f70f1665a825e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42373 } } peers { permanent_uuid: "c285c2aeffcc4c1893c721207b324e89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42363 } }
I20260927 10:47:39.601804 17895 heartbeater.cc:503] Master 127.0.98.62:45721 was elected leader, sending a full tablet report...
I20260927 10:47:39.601876 17914 ts_tablet_manager.cc:1428] T 029649a85680402eb91045d5ab8d4e8a P 587b34324562462d96146cec8e7e8ed3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:47:39.601876 17913 ts_tablet_manager.cc:1428] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:47:39.602751 17761 heartbeater.cc:503] Master 127.0.98.62:45721 was elected leader, sending a full tablet report...
W20260927 10:47:39.770792 17896 tablet.cc:2369] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:47:39.791858 17919 raft_consensus.cc:482] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:47:39.792050 17919 raft_consensus.cc:504] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587b34324562462d96146cec8e7e8ed3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34805 } } peers { permanent_uuid: "7fac15de23fb47ddaf2f70f1665a825e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42373 } } peers { permanent_uuid: "c285c2aeffcc4c1893c721207b324e89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42363 } }
I20260927 10:47:39.793308 17919 leader_election.cc:288] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 587b34324562462d96146cec8e7e8ed3 (127.0.98.2:34805), 7fac15de23fb47ddaf2f70f1665a825e (127.0.98.1:42373)
W20260927 10:47:39.795117 17762 tablet.cc:2369] T 029649a85680402eb91045d5ab8d4e8a P 587b34324562462d96146cec8e7e8ed3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:47:39.798197 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:39.793675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50523 (local address 127.0.98.1:42373)
0927 10:47:39.793840 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:39.793844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:39.793860 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:47:39.794049 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:39.794054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:39.794108 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:39.794193 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:39.794530 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.795316 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.797060 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.797650 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.797740 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:39.797799 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:39.798002 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 10:47:39.798074 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 10:47:39.798197 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:39.793728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57435 (local address 127.0.98.2:34805)
0927 10:47:39.793850 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:39.793855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:39.794305 (+ 450us) server_negotiation.cc:407] Connection header received
0927 10:47:39.794524 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:39.794527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:39.794575 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:39.794657 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:39.794925 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.795796 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.796924 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.797561 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.797676 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:39.797759 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:39.798002 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 10:47:39.798073 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:47:39.798221 17897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:39.793559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42373 (local address 127.0.98.3:50523)
0927 10:47:39.793762 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:39.793780 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:39.793896 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:39.794228 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:39.794231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:39.794253 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:39.794371 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:39.794377 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.795440 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.795443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:39.796926 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:39.796936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.797753 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.797756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:39.797878 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:39.797898 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:47:39.797972 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 10:47:39.798064 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":53}
I20260927 10:47:39.798230 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:39.793559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34805 (local address 127.0.98.3:57435)
0927 10:47:39.794221 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:39.794235 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:39.794377 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:39.794668 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:39.794671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:39.794686 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:39.794812 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:39.794817 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.795920 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.795925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:39.796789 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:39.796799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.797718 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.797724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:39.797829 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:39.797871 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 10:47:39.797958 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 10:47:39.798072 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":368,"thread_start_us":121,"threads_started":1}
I20260927 10:47:39.798918 17581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "029649a85680402eb91045d5ab8d4e8a" candidate_uuid: "c285c2aeffcc4c1893c721207b324e89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fac15de23fb47ddaf2f70f1665a825e" is_pre_election: true
I20260927 10:47:39.798916 17715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "029649a85680402eb91045d5ab8d4e8a" candidate_uuid: "c285c2aeffcc4c1893c721207b324e89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "587b34324562462d96146cec8e7e8ed3" is_pre_election: true
I20260927 10:47:39.799201 17581 raft_consensus.cc:2454] T 029649a85680402eb91045d5ab8d4e8a P 7fac15de23fb47ddaf2f70f1665a825e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c285c2aeffcc4c1893c721207b324e89 in term 0.
I20260927 10:47:39.799206 17715 raft_consensus.cc:2454] T 029649a85680402eb91045d5ab8d4e8a P 587b34324562462d96146cec8e7e8ed3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c285c2aeffcc4c1893c721207b324e89 in term 0.
I20260927 10:47:39.799692 17784 leader_election.cc:302] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 587b34324562462d96146cec8e7e8ed3, c285c2aeffcc4c1893c721207b324e89; no voters:
I20260927 10:47:39.799975 17919 raft_consensus.cc:2781] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:47:39.800055 17919 raft_consensus.cc:482] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:47:39.800086 17919 raft_consensus.cc:3037] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:39.801134 17919 raft_consensus.cc:504] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587b34324562462d96146cec8e7e8ed3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34805 } } peers { permanent_uuid: "7fac15de23fb47ddaf2f70f1665a825e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42373 } } peers { permanent_uuid: "c285c2aeffcc4c1893c721207b324e89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42363 } }
I20260927 10:47:39.801523 17919 leader_election.cc:288] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89 [CANDIDATE]: Term 1 election: Requested vote from peers 587b34324562462d96146cec8e7e8ed3 (127.0.98.2:34805), 7fac15de23fb47ddaf2f70f1665a825e (127.0.98.1:42373)
I20260927 10:47:39.801867 17581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "029649a85680402eb91045d5ab8d4e8a" candidate_uuid: "c285c2aeffcc4c1893c721207b324e89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fac15de23fb47ddaf2f70f1665a825e"
I20260927 10:47:39.801936 17715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "029649a85680402eb91045d5ab8d4e8a" candidate_uuid: "c285c2aeffcc4c1893c721207b324e89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "587b34324562462d96146cec8e7e8ed3"
I20260927 10:47:39.801982 17581 raft_consensus.cc:3037] T 029649a85680402eb91045d5ab8d4e8a P 7fac15de23fb47ddaf2f70f1665a825e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:39.802091 17715 raft_consensus.cc:3037] T 029649a85680402eb91045d5ab8d4e8a P 587b34324562462d96146cec8e7e8ed3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:39.803193 17581 raft_consensus.cc:2454] T 029649a85680402eb91045d5ab8d4e8a P 7fac15de23fb47ddaf2f70f1665a825e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c285c2aeffcc4c1893c721207b324e89 in term 1.
I20260927 10:47:39.803218 17715 raft_consensus.cc:2454] T 029649a85680402eb91045d5ab8d4e8a P 587b34324562462d96146cec8e7e8ed3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c285c2aeffcc4c1893c721207b324e89 in term 1.
I20260927 10:47:39.803519 17785 leader_election.cc:302] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fac15de23fb47ddaf2f70f1665a825e, c285c2aeffcc4c1893c721207b324e89; no voters:
I20260927 10:47:39.803711 17919 raft_consensus.cc:2781] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:39.803900 17919 raft_consensus.cc:686] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89 [term 1 LEADER]: Becoming Leader. State: Replica: c285c2aeffcc4c1893c721207b324e89, State: Running, Role: LEADER
I20260927 10:47:39.804247 17919 consensus_queue.cc:237] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587b34324562462d96146cec8e7e8ed3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34805 } } peers { permanent_uuid: "7fac15de23fb47ddaf2f70f1665a825e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42373 } } peers { permanent_uuid: "c285c2aeffcc4c1893c721207b324e89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42363 } }
I20260927 10:47:39.807689 17434 catalog_manager.cc:5495] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89 reported cstate change: term changed from 0 to 1, leader changed from <none> to c285c2aeffcc4c1893c721207b324e89 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c285c2aeffcc4c1893c721207b324e89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587b34324562462d96146cec8e7e8ed3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fac15de23fb47ddaf2f70f1665a825e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c285c2aeffcc4c1893c721207b324e89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42363 } health_report { overall_health: HEALTHY } } }
W20260927 10:47:39.827049 17628 tablet.cc:2369] T 029649a85680402eb91045d5ab8d4e8a P 7fac15de23fb47ddaf2f70f1665a825e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:47:39.872802 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:39.869515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52806 (local address 127.0.98.3:42363)
0927 10:47:39.869644 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:39.869648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:39.869661 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:47:39.869766 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:39.869769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:39.869826 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:39.869922 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:47:39.870131 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.870691 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.871540 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:39.871736 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:39.871806 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:39.872073 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:47:39.872481 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:47:39.872569 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:39.872627 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 10:47:39.872675 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:47:39.878497 17581 raft_consensus.cc:1261] T 029649a85680402eb91045d5ab8d4e8a P 7fac15de23fb47ddaf2f70f1665a825e [term 1 FOLLOWER]: Refusing update from remote peer c285c2aeffcc4c1893c721207b324e89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:47:39.878510 17715 raft_consensus.cc:1261] T 029649a85680402eb91045d5ab8d4e8a P 587b34324562462d96146cec8e7e8ed3 [term 1 FOLLOWER]: Refusing update from remote peer c285c2aeffcc4c1893c721207b324e89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:47:39.879158 17919 consensus_queue.cc:1035] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fac15de23fb47ddaf2f70f1665a825e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:47:39.879415 17922 consensus_queue.cc:1035] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "587b34324562462d96146cec8e7e8ed3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:47:39.889932 17927 mvcc.cc:205] Tried to move back new op lower bound from 7333912821255041024 to 7333912820960579584. Current Snapshot: MvccSnapshot[applied={T|T < 7333912821255041024}]
I20260927 10:47:39.895429 17435 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:47:39.895581 17435 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:47:39.898027 17435 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3675aa08ab04b669b604e9190f8a767: Sending DeleteTablet for 3 replicas of tablet 029649a85680402eb91045d5ab8d4e8a
I20260927 10:47:39.898864 17829 tablet_service.cc:1514] Processing DeleteTablet for tablet 029649a85680402eb91045d5ab8d4e8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52790
I20260927 10:47:39.898846 17695 tablet_service.cc:1514] Processing DeleteTablet for tablet 029649a85680402eb91045d5ab8d4e8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37994
I20260927 10:47:39.899381 17934 tablet_replica.cc:330] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89: stopping tablet replica
I20260927 10:47:39.899582 17561 tablet_service.cc:1514] Processing DeleteTablet for tablet 029649a85680402eb91045d5ab8d4e8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49892
I20260927 10:47:39.899744 17934 raft_consensus.cc:2229] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:47:39.900064 17936 tablet_replica.cc:330] T 029649a85680402eb91045d5ab8d4e8a P 7fac15de23fb47ddaf2f70f1665a825e: stopping tablet replica
I20260927 10:47:39.900106 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 17498
I20260927 10:47:39.900365 17936 raft_consensus.cc:2229] T 029649a85680402eb91045d5ab8d4e8a P 7fac15de23fb47ddaf2f70f1665a825e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:47:39.900180 17934 raft_consensus.cc:2258] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:39.902673 17935 tablet_replica.cc:330] T 029649a85680402eb91045d5ab8d4e8a P 587b34324562462d96146cec8e7e8ed3: stopping tablet replica
I20260927 10:47:39.903007 17935 raft_consensus.cc:2229] T 029649a85680402eb91045d5ab8d4e8a P 587b34324562462d96146cec8e7e8ed3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:47:39.903360 17935 raft_consensus.cc:2258] T 029649a85680402eb91045d5ab8d4e8a P 587b34324562462d96146cec8e7e8ed3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:39.903885 17934 ts_tablet_manager.cc:1905] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:47:39.904827 17935 ts_tablet_manager.cc:1905] T 029649a85680402eb91045d5ab8d4e8a P 587b34324562462d96146cec8e7e8ed3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:47:39.907629 17935 ts_tablet_manager.cc:1918] T 029649a85680402eb91045d5ab8d4e8a P 587b34324562462d96146cec8e7e8ed3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:47:39.907683 17934 ts_tablet_manager.cc:1918] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:47:39.907764 17935 log.cc:1196] T 029649a85680402eb91045d5ab8d4e8a P 587b34324562462d96146cec8e7e8ed3: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-1/wal/wals/029649a85680402eb91045d5ab8d4e8a
I20260927 10:47:39.907915 17934 log.cc:1196] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/029649a85680402eb91045d5ab8d4e8a
I20260927 10:47:39.908121 17935 ts_tablet_manager.cc:1939] T 029649a85680402eb91045d5ab8d4e8a P 587b34324562462d96146cec8e7e8ed3: Deleting consensus metadata
W20260927 10:47:39.908170 17423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42373 (error 108)
I20260927 10:47:39.908445 17934 ts_tablet_manager.cc:1939] T 029649a85680402eb91045d5ab8d4e8a P c285c2aeffcc4c1893c721207b324e89: Deleting consensus metadata
I20260927 10:47:39.909076 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 17632
I20260927 10:47:39.911056 17423 catalog_manager.cc:4841] TS c285c2aeffcc4c1893c721207b324e89 (127.0.98.3:42363): tablet 029649a85680402eb91045d5ab8d4e8a (table test-table [id=14caea946b264a5bb20163f085e03653]) successfully deleted
I20260927 10:47:39.911181 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:39.910874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42373 (local address 127.0.0.1:49904)
0927 10:47:39.911015 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:39.911096 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 10:47:39.911334 17423 catalog_manager.cc:4584] TS 7fac15de23fb47ddaf2f70f1665a825e (127.0.98.1:42373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 029649a85680402eb91045d5ab8d4e8a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42373: connect: Connection refused (error 111)
I20260927 10:47:39.915570 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 17766
I20260927 10:47:39.916739 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:39.915839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34805 (local address 127.0.0.1:38008)
0927 10:47:39.916574 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:39.916636 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":634}
I20260927 10:47:39.921298 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 17404
2026-09-27T10:47:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2491 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T10:47:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:47:39Z Disabled control of system clock
I20260927 10:47:39.951993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43469
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:47:40.061725 17941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:40.062011 17941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:40.062048 17941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:40.065979 17941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:40.066038 17941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:40.066054 17941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:40.066072 17941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:40.066087 17941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:40.070744 17941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43469
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:40.071779 17941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:40.073107 17941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:40.081643 17949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:40.081718 17946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:40.081652 17947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:40.081693 17941 server_base.cc:1034] running on GCE node
I20260927 10:47:40.082348 17941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:40.083531 17941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:40.084738 17941 hybrid_clock.cc:648] HybridClock initialized: now 1790506060084718 us; error 35 us; skew 500 ppm
I20260927 10:47:40.086769 17941 webserver.cc:460] Webserver started at http://127.0.98.62:38681/ using document root <none> and password file <none>
I20260927 10:47:40.087412 17941 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:40.087505 17941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:40.087733 17941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:40.089766 17941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "2f51aef3e88f4ce2898d7621637f3501"
format_stamp: "Formatted at 2026-09-27 10:47:40 on dist-test-slave-dzhw"
server_key: "18eea061fe2c031b4420bd9511b2618c"
server_key_iv: "83d2fcac605b55fbb845434f013eedab"
server_key_version: "encryptionkey@0"
I20260927 10:47:40.090332 17941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "2f51aef3e88f4ce2898d7621637f3501"
format_stamp: "Formatted at 2026-09-27 10:47:40 on dist-test-slave-dzhw"
server_key: "18eea061fe2c031b4420bd9511b2618c"
server_key_iv: "83d2fcac605b55fbb845434f013eedab"
server_key_version: "encryptionkey@0"
I20260927 10:47:40.093986 17941 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:47:40.096601 17955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:40.097873 17941 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:47:40.098016 17941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "2f51aef3e88f4ce2898d7621637f3501"
format_stamp: "Formatted at 2026-09-27 10:47:40 on dist-test-slave-dzhw"
server_key: "18eea061fe2c031b4420bd9511b2618c"
server_key_iv: "83d2fcac605b55fbb845434f013eedab"
server_key_version: "encryptionkey@0"
I20260927 10:47:40.098140 17941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:40.126027 17941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:40.126816 17941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:40.127028 17941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:40.134065 17941 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43469
I20260927 10:47:40.134073 18007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43469 every 8 connection(s)
I20260927 10:47:40.135039 17941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:47:40.138111 18008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:40.138105 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 17941
I20260927 10:47:40.138268 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:47:40.138629 392 external_mini_cluster.cc:1428] Setting key 32c48a4bd40629316e0a97bf3b984ba6
I20260927 10:47:40.144706 18008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f51aef3e88f4ce2898d7621637f3501: Bootstrap starting.
I20260927 10:47:40.147070 18008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f51aef3e88f4ce2898d7621637f3501: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:40.147841 18008 log.cc:824] T 00000000000000000000000000000000 P 2f51aef3e88f4ce2898d7621637f3501: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:40.147876 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:40.139997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59288 (local address 127.0.98.62:43469)
0927 10:47:40.140624 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:40.140639 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:40.140673 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:47:40.141378 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:40.141395 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:40.141767 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:40.142103 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:40.142593 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.143566 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.144442 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.144954 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.145113 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:40.146023 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:40.146028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:40.146042 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:47:40.146522 (+ 480us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:40.146580 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:47:40.146601 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:40.146703 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:40.146860 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 10:47:40.147128 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":290,"thread_start_us":134,"threads_started":1}
I20260927 10:47:40.150063 18008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f51aef3e88f4ce2898d7621637f3501: No bootstrap required, opened a new log
I20260927 10:47:40.153590 18008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f51aef3e88f4ce2898d7621637f3501 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f51aef3e88f4ce2898d7621637f3501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } }
I20260927 10:47:40.153847 18008 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f51aef3e88f4ce2898d7621637f3501 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:40.153882 18008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f51aef3e88f4ce2898d7621637f3501 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f51aef3e88f4ce2898d7621637f3501, State: Initialized, Role: FOLLOWER
I20260927 10:47:40.154250 18008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f51aef3e88f4ce2898d7621637f3501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f51aef3e88f4ce2898d7621637f3501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } }
I20260927 10:47:40.154475 18008 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f51aef3e88f4ce2898d7621637f3501 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:47:40.154536 18008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f51aef3e88f4ce2898d7621637f3501 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:47:40.154611 18008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f51aef3e88f4ce2898d7621637f3501 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:40.155642 18008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f51aef3e88f4ce2898d7621637f3501 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f51aef3e88f4ce2898d7621637f3501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } }
I20260927 10:47:40.156050 18008 leader_election.cc:302] T 00000000000000000000000000000000 P 2f51aef3e88f4ce2898d7621637f3501 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f51aef3e88f4ce2898d7621637f3501; no voters:
I20260927 10:47:40.156421 18008 leader_election.cc:288] T 00000000000000000000000000000000 P 2f51aef3e88f4ce2898d7621637f3501 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:47:40.156553 18013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f51aef3e88f4ce2898d7621637f3501 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:40.156810 18013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f51aef3e88f4ce2898d7621637f3501 [term 1 LEADER]: Becoming Leader. State: Replica: 2f51aef3e88f4ce2898d7621637f3501, State: Running, Role: LEADER
I20260927 10:47:40.157138 18013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f51aef3e88f4ce2898d7621637f3501 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f51aef3e88f4ce2898d7621637f3501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } }
I20260927 10:47:40.157704 18008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f51aef3e88f4ce2898d7621637f3501 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:47:40.158908 18014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f51aef3e88f4ce2898d7621637f3501 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f51aef3e88f4ce2898d7621637f3501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f51aef3e88f4ce2898d7621637f3501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } } }
I20260927 10:47:40.159035 18014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f51aef3e88f4ce2898d7621637f3501 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:40.159372 18015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f51aef3e88f4ce2898d7621637f3501 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f51aef3e88f4ce2898d7621637f3501. Latest consensus state: current_term: 1 leader_uuid: "2f51aef3e88f4ce2898d7621637f3501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f51aef3e88f4ce2898d7621637f3501" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } } }
I20260927 10:47:40.159451 18022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:47:40.159571 18015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f51aef3e88f4ce2898d7621637f3501 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:40.162320 18022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:47:40.167884 18022 catalog_manager.cc:1347] Generated new cluster ID: 4d6a6cdca6c74475a518dc5522db9421
I20260927 10:47:40.167959 18022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:47:40.190188 18022 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:47:40.191455 18022 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:47:40.201062 18022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f51aef3e88f4ce2898d7621637f3501: Generated new TSK 0
I20260927 10:47:40.201882 18022 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:47:40.213500 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43469
--builtin_ntp_servers=127.0.98.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:47:40.325546 18032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:40.325822 18032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:40.325862 18032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:40.329876 18032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:40.329941 18032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:40.329996 18032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:47:40.334743 18032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:40.335803 18032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:40.337096 18032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:40.346740 18038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:40.346838 18032 server_base.cc:1034] running on GCE node
W20260927 10:47:40.346760 18037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:40.346733 18040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:40.347375 18032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:40.348098 18032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:40.349310 18032 hybrid_clock.cc:648] HybridClock initialized: now 1790506060349271 us; error 51 us; skew 500 ppm
I20260927 10:47:40.351323 18032 webserver.cc:460] Webserver started at http://127.0.98.1:35817/ using document root <none> and password file <none>
I20260927 10:47:40.351980 18032 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:40.352083 18032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:40.352316 18032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:40.354492 18032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "490d2f8453b7471cbcb5b45fc1161674"
format_stamp: "Formatted at 2026-09-27 10:47:40 on dist-test-slave-dzhw"
server_key: "ededf015236b1093f1e7633502cff9d9"
server_key_iv: "16afc48bdf388898803b439caef1275f"
server_key_version: "encryptionkey@0"
I20260927 10:47:40.355064 18032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "490d2f8453b7471cbcb5b45fc1161674"
format_stamp: "Formatted at 2026-09-27 10:47:40 on dist-test-slave-dzhw"
server_key: "ededf015236b1093f1e7633502cff9d9"
server_key_iv: "16afc48bdf388898803b439caef1275f"
server_key_version: "encryptionkey@0"
I20260927 10:47:40.359426 18032 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 10:47:40.362551 18046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:40.363916 18032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:47:40.364027 18032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "490d2f8453b7471cbcb5b45fc1161674"
format_stamp: "Formatted at 2026-09-27 10:47:40 on dist-test-slave-dzhw"
server_key: "ededf015236b1093f1e7633502cff9d9"
server_key_iv: "16afc48bdf388898803b439caef1275f"
server_key_version: "encryptionkey@0"
I20260927 10:47:40.364127 18032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:40.388922 18032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:40.389695 18032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:40.389853 18032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:40.390403 18032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:40.391410 18032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:40.391469 18032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:40.391510 18032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:40.391525 18032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:40.401628 18032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33671
I20260927 10:47:40.401697 18159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33671 every 8 connection(s)
I20260927 10:47:40.402801 18032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:47:40.410167 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 18032
I20260927 10:47:40.410279 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:47:40.410632 392 external_mini_cluster.cc:1428] Setting key c7c7da3f09413ab9dbcd491f28e5d3f3
I20260927 10:47:40.412777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43469
--builtin_ntp_servers=127.0.98.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:47:40.413290 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:40.404792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46483 (local address 127.0.98.62:43469)
0927 10:47:40.404937 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:40.404940 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:40.406448 (+ 1508us) server_negotiation.cc:407] Connection header received
0927 10:47:40.407906 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:40.407909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:40.407972 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:40.408092 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:40.409024 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.409602 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.410999 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.411288 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.411400 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:40.412751 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:40.412755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:40.412757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:47:40.412771 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:40.412798 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:47:40.412807 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:40.412894 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:40.413118 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 10:47:40.413175 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:47:40.414561 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:40.405345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43469 (local address 127.0.98.1:46483)
0927 10:47:40.406226 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:40.406275 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:40.407678 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:40.408266 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:40.408274 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:40.408359 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:40.408872 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:40.408885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.409752 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.409755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:40.410865 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:40.410875 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.411484 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.411489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:40.411692 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:40.412458 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:47:40.412460 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:40.412503 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:47:40.412588 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:40.412929 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:40.412943 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:40.412953 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 10:47:40.413104 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 10:47:40.413447 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":148,"threads_started":1}
I20260927 10:47:40.415849 18160 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43469
I20260927 10:47:40.416097 18160 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:40.416690 18160 heartbeater.cc:511] Master 127.0.98.62:43469 requested a full tablet report, sending...
I20260927 10:47:40.418148 17972 ts_manager.cc:194] Registered new tserver with Master: 490d2f8453b7471cbcb5b45fc1161674 (127.0.98.1:33671)
I20260927 10:47:40.419349 17972 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:46483
W20260927 10:47:40.522738 18163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:40.523061 18163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:40.523131 18163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:40.527186 18163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:40.527288 18163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:40.527384 18163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:47:40.532354 18163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:40.533550 18163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:40.535022 18163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:47:40.544997 18163 server_base.cc:1034] running on GCE node
W20260927 10:47:40.544927 18171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:40.544927 18169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:40.544945 18168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:40.545622 18163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:40.546329 18163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:40.547523 18163 hybrid_clock.cc:648] HybridClock initialized: now 1790506060547492 us; error 47 us; skew 500 ppm
I20260927 10:47:40.549466 18163 webserver.cc:460] Webserver started at http://127.0.98.2:41361/ using document root <none> and password file <none>
I20260927 10:47:40.550062 18163 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:40.550163 18163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:40.550391 18163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:40.552518 18163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "ce5eeb75987d40edace626f6e4ef127a"
format_stamp: "Formatted at 2026-09-27 10:47:40 on dist-test-slave-dzhw"
server_key: "d9aa3c29a51dc71168f69b91e1188879"
server_key_iv: "c85c0cccc3c4e8f76216bb1b6e90f06c"
server_key_version: "encryptionkey@0"
I20260927 10:47:40.553022 18163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ce5eeb75987d40edace626f6e4ef127a"
format_stamp: "Formatted at 2026-09-27 10:47:40 on dist-test-slave-dzhw"
server_key: "d9aa3c29a51dc71168f69b91e1188879"
server_key_iv: "c85c0cccc3c4e8f76216bb1b6e90f06c"
server_key_version: "encryptionkey@0"
I20260927 10:47:40.556725 18163 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:47:40.559721 18177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:40.561092 18163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:47:40.561235 18163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "ce5eeb75987d40edace626f6e4ef127a"
format_stamp: "Formatted at 2026-09-27 10:47:40 on dist-test-slave-dzhw"
server_key: "d9aa3c29a51dc71168f69b91e1188879"
server_key_iv: "c85c0cccc3c4e8f76216bb1b6e90f06c"
server_key_version: "encryptionkey@0"
I20260927 10:47:40.561360 18163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:40.576221 18163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:40.576965 18163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:40.577173 18163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:40.577771 18163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:40.578778 18163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:40.578848 18163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:40.578914 18163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:40.578964 18163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:40.588055 18163 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44935
I20260927 10:47:40.588080 18290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44935 every 8 connection(s)
I20260927 10:47:40.589190 18163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:47:40.589735 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 18163
I20260927 10:47:40.589874 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:47:40.590210 392 external_mini_cluster.cc:1428] Setting key f38016038f37ed3b42dcb1bbcb32a253
I20260927 10:47:40.592614 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43469
--builtin_ntp_servers=127.0.98.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:47:40.600692 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:40.591213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49093 (local address 127.0.98.62:43469)
0927 10:47:40.591352 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:40.591357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:40.592744 (+ 1387us) server_negotiation.cc:407] Connection header received
0927 10:47:40.594782 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:40.594788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:40.594855 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:40.594961 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:40.595852 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.596790 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.598111 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.598588 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.598724 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:40.599876 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:40.599882 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:40.599888 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:47:40.599907 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:40.599934 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:47:40.599946 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:40.600047 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:40.600263 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 10:47:40.600347 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:47:40.601464 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:40.591774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43469 (local address 127.0.98.2:49093)
0927 10:47:40.592561 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:40.592618 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:40.594544 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:40.595119 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:40.595126 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:40.595215 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:40.595702 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:40.595714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.596924 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.596927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:40.597981 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:40.597990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.598786 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.598791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:40.598947 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:40.599599 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:47:40.599600 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:40.599641 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:47:40.599708 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:40.600086 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:40.600095 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:40.600100 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 10:47:40.600232 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 10:47:40.600492 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":122,"threads_started":1}
I20260927 10:47:40.602717 18291 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43469
I20260927 10:47:40.602958 18291 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:40.603485 18291 heartbeater.cc:511] Master 127.0.98.62:43469 requested a full tablet report, sending...
I20260927 10:47:40.604562 17972 ts_manager.cc:194] Registered new tserver with Master: ce5eeb75987d40edace626f6e4ef127a (127.0.98.2:44935)
I20260927 10:47:40.605165 17972 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:49093
W20260927 10:47:40.706954 18294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:40.707235 18294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:40.707274 18294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:40.711465 18294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:40.711529 18294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:40.711611 18294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:47:40.716450 18294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:40.717523 18294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:40.718968 18294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:40.728972 18299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:40.728972 18302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:40.729075 18300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:40.729221 18294 server_base.cc:1034] running on GCE node
I20260927 10:47:40.729776 18294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:40.730460 18294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:40.731642 18294 hybrid_clock.cc:648] HybridClock initialized: now 1790506060731624 us; error 29 us; skew 500 ppm
I20260927 10:47:40.733448 18294 webserver.cc:460] Webserver started at http://127.0.98.3:44237/ using document root <none> and password file <none>
I20260927 10:47:40.733983 18294 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:40.734043 18294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:40.734211 18294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:40.736266 18294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "f5f4d448414e42da9c8a007328b459ab"
format_stamp: "Formatted at 2026-09-27 10:47:40 on dist-test-slave-dzhw"
server_key: "5b95a68cabff5e82fe4e5cec72cbc430"
server_key_iv: "bf475ce36c9dd6a724084441d8ce511e"
server_key_version: "encryptionkey@0"
I20260927 10:47:40.736799 18294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f5f4d448414e42da9c8a007328b459ab"
format_stamp: "Formatted at 2026-09-27 10:47:40 on dist-test-slave-dzhw"
server_key: "5b95a68cabff5e82fe4e5cec72cbc430"
server_key_iv: "bf475ce36c9dd6a724084441d8ce511e"
server_key_version: "encryptionkey@0"
I20260927 10:47:40.740509 18294 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:47:40.742985 18308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:40.744226 18294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:47:40.744357 18294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "f5f4d448414e42da9c8a007328b459ab"
format_stamp: "Formatted at 2026-09-27 10:47:40 on dist-test-slave-dzhw"
server_key: "5b95a68cabff5e82fe4e5cec72cbc430"
server_key_iv: "bf475ce36c9dd6a724084441d8ce511e"
server_key_version: "encryptionkey@0"
I20260927 10:47:40.744465 18294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:40.754851 18294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:40.755503 18294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:40.755653 18294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:40.756170 18294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:40.757054 18294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:40.757100 18294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:40.757140 18294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:40.757155 18294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:40.766649 18294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40997
I20260927 10:47:40.766671 18421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40997 every 8 connection(s)
I20260927 10:47:40.767647 18294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:47:40.769497 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 18294
I20260927 10:47:40.769601 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:47:40.769910 392 external_mini_cluster.cc:1428] Setting key 71bf8ca681d574a8d46476c658e1ee1a
I20260927 10:47:40.777160 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:40.769674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50657 (local address 127.0.98.62:43469)
0927 10:47:40.769820 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:40.769824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:40.771242 (+ 1418us) server_negotiation.cc:407] Connection header received
0927 10:47:40.772816 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:40.772818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:40.772863 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:40.772930 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:40.773774 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.774297 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.775491 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.775701 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.775783 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:40.776762 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:40.776765 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:40.776767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:47:40.776779 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:40.776796 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:47:40.776804 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:40.776861 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:40.776998 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 10:47:40.777044 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:47:40.778051 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:40.770207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43469 (local address 127.0.98.3:50657)
0927 10:47:40.771073 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:40.771133 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:40.772633 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:40.773060 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:40.773066 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:40.773146 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:40.773636 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:40.773648 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.774401 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.774404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:40.775360 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:40.775369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.775829 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.775832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:40.775966 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:40.776526 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:47:40.776528 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:40.776566 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:47:40.776629 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:40.776877 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:40.776882 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:40.776886 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 10:47:40.776981 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 10:47:40.777195 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":202,"threads_started":1}
I20260927 10:47:40.779202 18422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43469
I20260927 10:47:40.779462 18422 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:40.779991 18422 heartbeater.cc:511] Master 127.0.98.62:43469 requested a full tablet report, sending...
I20260927 10:47:40.780990 17972 ts_manager.cc:194] Registered new tserver with Master: f5f4d448414e42da9c8a007328b459ab (127.0.98.3:40997)
I20260927 10:47:40.781574 17972 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:50657
I20260927 10:47:40.783903 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:47:40.789362 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:40.785833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59296 (local address 127.0.98.62:43469)
0927 10:47:40.785950 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:40.785955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:40.786112 (+ 157us) server_negotiation.cc:407] Connection header received
0927 10:47:40.786203 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:40.786205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:40.786243 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:40.786309 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:40.786631 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.787120 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.787981 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.788190 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.788276 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:40.789033 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:40.789036 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:40.789038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:47:40.789049 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:40.789067 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:47:40.789075 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:40.789134 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:40.789207 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 10:47:40.789252 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:47:40.792374 17972 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:47:40.795447 17972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:47:40.810184 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:40.805023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33671 (local address 127.0.0.1:50670)
0927 10:47:40.805498 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:40.805513 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:40.805592 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:40.806166 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:40.806169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:40.806182 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:40.806249 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:40.806253 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.807670 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.807674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:40.808874 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:40.808882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.809850 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.809855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:40.809945 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:40.809964 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 10:47:40.810021 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 10:47:40.810075 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":24,"thread_start_us":85,"threads_started":1}
I20260927 10:47:40.810343 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:40.805302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50670 (local address 127.0.98.1:33671)
0927 10:47:40.805748 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:40.805759 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:40.805781 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:47:40.805856 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:40.805863 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:40.806028 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:40.806192 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:40.806335 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.807530 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.809002 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.809724 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.809830 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:40.810011 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:40.810065 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 10:47:40.810168 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":126,"threads_started":1}
I20260927 10:47:40.810840 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:40.804157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55042 (local address 127.0.98.3:40997)
0927 10:47:40.804670 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:40.804684 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:40.804711 (+ 27us) server_negotiation.cc:407] Connection header received
0927 10:47:40.804788 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:40.804793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:40.804997 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:40.805156 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:40.805692 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.806722 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.808611 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.809431 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.809555 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:40.809772 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:40.810249 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 10:47:40.810649 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":123,"threads_started":1}
I20260927 10:47:40.810931 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:40.804706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44935 (local address 127.0.0.1:40698)
0927 10:47:40.805265 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:40.805279 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:40.805403 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:40.805927 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:40.805931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:40.805950 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:40.806039 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:40.806045 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.808151 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.808155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:40.809192 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:40.809203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.810506 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.810510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:40.810610 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:40.810635 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:47:40.810707 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 10:47:40.810790 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":25,"spinlock_wait_cycles":2304,"thread_start_us":90,"threads_started":1}
I20260927 10:47:40.811378 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:40.804911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40698 (local address 127.0.98.2:44935)
0927 10:47:40.805342 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:40.805353 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:40.805376 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:47:40.805579 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:40.805585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:40.805769 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:40.805918 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:40.806161 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.808014 (+ 1853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.809327 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.810130 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.810232 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:40.810406 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:40.811104 (+ 698us) server_negotiation.cc:299] Negotiation successful
0927 10:47:40.811210 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":137,"threads_started":1}
I20260927 10:47:40.810184 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:40.803996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40997 (local address 127.0.0.1:55042)
0927 10:47:40.804365 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:40.804417 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:40.804553 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:40.805351 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:40.805366 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:40.805404 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:40.805555 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:40.805565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.806828 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.806833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:40.808469 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:40.808480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.809558 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.809562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:40.809681 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:40.809719 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 10:47:40.809846 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 10:47:40.810005 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":1,"spinlock_wait_cycles":3840,"thread_start_us":79,"threads_started":1}
I20260927 10:47:40.813884 18094 tablet_service.cc:1467] Processing CreateTablet for tablet 9c53a1d7f9b74290b7071d8834d795c0 (DEFAULT_TABLE table=test-table [id=a81288b69e4949bba7a327da263b6ff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:40.814122 18225 tablet_service.cc:1467] Processing CreateTablet for tablet 9c53a1d7f9b74290b7071d8834d795c0 (DEFAULT_TABLE table=test-table [id=a81288b69e4949bba7a327da263b6ff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:40.814136 18356 tablet_service.cc:1467] Processing CreateTablet for tablet 9c53a1d7f9b74290b7071d8834d795c0 (DEFAULT_TABLE table=test-table [id=a81288b69e4949bba7a327da263b6ff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:40.815155 18356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c53a1d7f9b74290b7071d8834d795c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:40.815155 18225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c53a1d7f9b74290b7071d8834d795c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:40.815155 18094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c53a1d7f9b74290b7071d8834d795c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:40.821527 18439 tablet_bootstrap.cc:492] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674: Bootstrap starting.
I20260927 10:47:40.822773 18440 tablet_bootstrap.cc:492] T 9c53a1d7f9b74290b7071d8834d795c0 P ce5eeb75987d40edace626f6e4ef127a: Bootstrap starting.
I20260927 10:47:40.823801 18439 tablet_bootstrap.cc:654] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:40.824580 18439 log.cc:824] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:40.824615 18440 tablet_bootstrap.cc:654] T 9c53a1d7f9b74290b7071d8834d795c0 P ce5eeb75987d40edace626f6e4ef127a: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:40.825384 18441 tablet_bootstrap.cc:492] T 9c53a1d7f9b74290b7071d8834d795c0 P f5f4d448414e42da9c8a007328b459ab: Bootstrap starting.
I20260927 10:47:40.825519 18440 log.cc:824] T 9c53a1d7f9b74290b7071d8834d795c0 P ce5eeb75987d40edace626f6e4ef127a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:40.827658 18440 tablet_bootstrap.cc:492] T 9c53a1d7f9b74290b7071d8834d795c0 P ce5eeb75987d40edace626f6e4ef127a: No bootstrap required, opened a new log
I20260927 10:47:40.827658 18439 tablet_bootstrap.cc:492] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674: No bootstrap required, opened a new log
I20260927 10:47:40.827787 18441 tablet_bootstrap.cc:654] T 9c53a1d7f9b74290b7071d8834d795c0 P f5f4d448414e42da9c8a007328b459ab: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:40.827872 18439 ts_tablet_manager.cc:1397] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260927 10:47:40.827862 18440 ts_tablet_manager.cc:1397] T 9c53a1d7f9b74290b7071d8834d795c0 P ce5eeb75987d40edace626f6e4ef127a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:47:40.828502 18441 log.cc:824] T 9c53a1d7f9b74290b7071d8834d795c0 P f5f4d448414e42da9c8a007328b459ab: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:40.830199 18441 tablet_bootstrap.cc:492] T 9c53a1d7f9b74290b7071d8834d795c0 P f5f4d448414e42da9c8a007328b459ab: No bootstrap required, opened a new log
I20260927 10:47:40.830403 18441 ts_tablet_manager.cc:1397] T 9c53a1d7f9b74290b7071d8834d795c0 P f5f4d448414e42da9c8a007328b459ab: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:47:40.832360 18440 raft_consensus.cc:348] T 9c53a1d7f9b74290b7071d8834d795c0 P ce5eeb75987d40edace626f6e4ef127a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f4d448414e42da9c8a007328b459ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40997 } } peers { permanent_uuid: "ce5eeb75987d40edace626f6e4ef127a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44935 } } peers { permanent_uuid: "490d2f8453b7471cbcb5b45fc1161674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33671 } }
I20260927 10:47:40.832353 18439 raft_consensus.cc:348] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f4d448414e42da9c8a007328b459ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40997 } } peers { permanent_uuid: "ce5eeb75987d40edace626f6e4ef127a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44935 } } peers { permanent_uuid: "490d2f8453b7471cbcb5b45fc1161674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33671 } }
I20260927 10:47:40.832648 18440 raft_consensus.cc:374] T 9c53a1d7f9b74290b7071d8834d795c0 P ce5eeb75987d40edace626f6e4ef127a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:40.832649 18439 raft_consensus.cc:374] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:40.832700 18440 raft_consensus.cc:729] T 9c53a1d7f9b74290b7071d8834d795c0 P ce5eeb75987d40edace626f6e4ef127a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce5eeb75987d40edace626f6e4ef127a, State: Initialized, Role: FOLLOWER
I20260927 10:47:40.832700 18439 raft_consensus.cc:729] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 490d2f8453b7471cbcb5b45fc1161674, State: Initialized, Role: FOLLOWER
I20260927 10:47:40.833176 18440 consensus_queue.cc:260] T 9c53a1d7f9b74290b7071d8834d795c0 P ce5eeb75987d40edace626f6e4ef127a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f4d448414e42da9c8a007328b459ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40997 } } peers { permanent_uuid: "ce5eeb75987d40edace626f6e4ef127a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44935 } } peers { permanent_uuid: "490d2f8453b7471cbcb5b45fc1161674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33671 } }
I20260927 10:47:40.833176 18439 consensus_queue.cc:260] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f4d448414e42da9c8a007328b459ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40997 } } peers { permanent_uuid: "ce5eeb75987d40edace626f6e4ef127a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44935 } } peers { permanent_uuid: "490d2f8453b7471cbcb5b45fc1161674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33671 } }
I20260927 10:47:40.834230 18440 ts_tablet_manager.cc:1428] T 9c53a1d7f9b74290b7071d8834d795c0 P ce5eeb75987d40edace626f6e4ef127a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:47:40.834230 18439 ts_tablet_manager.cc:1428] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:47:40.834631 18160 heartbeater.cc:503] Master 127.0.98.62:43469 was elected leader, sending a full tablet report...
I20260927 10:47:40.834635 18291 heartbeater.cc:503] Master 127.0.98.62:43469 was elected leader, sending a full tablet report...
I20260927 10:47:40.834233 18441 raft_consensus.cc:348] T 9c53a1d7f9b74290b7071d8834d795c0 P f5f4d448414e42da9c8a007328b459ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f4d448414e42da9c8a007328b459ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40997 } } peers { permanent_uuid: "ce5eeb75987d40edace626f6e4ef127a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44935 } } peers { permanent_uuid: "490d2f8453b7471cbcb5b45fc1161674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33671 } }
I20260927 10:47:40.835533 18441 raft_consensus.cc:374] T 9c53a1d7f9b74290b7071d8834d795c0 P f5f4d448414e42da9c8a007328b459ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:40.835613 18441 raft_consensus.cc:729] T 9c53a1d7f9b74290b7071d8834d795c0 P f5f4d448414e42da9c8a007328b459ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5f4d448414e42da9c8a007328b459ab, State: Initialized, Role: FOLLOWER
I20260927 10:47:40.837378 18446 raft_consensus.cc:482] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:47:40.837529 18446 raft_consensus.cc:504] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f4d448414e42da9c8a007328b459ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40997 } } peers { permanent_uuid: "ce5eeb75987d40edace626f6e4ef127a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44935 } } peers { permanent_uuid: "490d2f8453b7471cbcb5b45fc1161674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33671 } }
I20260927 10:47:40.837678 18441 consensus_queue.cc:260] T 9c53a1d7f9b74290b7071d8834d795c0 P f5f4d448414e42da9c8a007328b459ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f4d448414e42da9c8a007328b459ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40997 } } peers { permanent_uuid: "ce5eeb75987d40edace626f6e4ef127a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44935 } } peers { permanent_uuid: "490d2f8453b7471cbcb5b45fc1161674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33671 } }
I20260927 10:47:40.839195 18441 ts_tablet_manager.cc:1428] T 9c53a1d7f9b74290b7071d8834d795c0 P f5f4d448414e42da9c8a007328b459ab: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260927 10:47:40.839411 18422 heartbeater.cc:503] Master 127.0.98.62:43469 was elected leader, sending a full tablet report...
W20260927 10:47:40.840247 18292 tablet.cc:2369] T 9c53a1d7f9b74290b7071d8834d795c0 P ce5eeb75987d40edace626f6e4ef127a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:47:40.840294 18446 leader_election.cc:288] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5f4d448414e42da9c8a007328b459ab (127.0.98.3:40997), ce5eeb75987d40edace626f6e4ef127a (127.0.98.2:44935)
I20260927 10:47:40.843070 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:40.838827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40997 (local address 127.0.98.1:44295)
0927 10:47:40.839091 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:40.839109 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:40.839192 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:40.839500 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:40.839503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:40.839520 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:40.839642 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:40.839650 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.840632 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.840635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:40.841792 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:40.841805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.842675 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.842678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:40.842752 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:40.842769 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 10:47:40.842871 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 10:47:40.842945 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260927 10:47:40.843070 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:40.839060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44295 (local address 127.0.98.3:40997)
0927 10:47:40.839158 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:40.839163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:40.839180 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:47:40.839308 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:40.839311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:40.839381 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:40.839453 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:40.839771 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.840485 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.841939 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.842414 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.842554 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:40.842613 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:40.842887 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 10:47:40.842950 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28,"spinlock_wait_cycles":13696}
I20260927 10:47:40.844259 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:40.839568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44935 (local address 127.0.98.1:34849)
0927 10:47:40.839934 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:40.839949 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:40.840028 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:40.840864 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:40.840866 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:40.840882 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:40.840981 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:40.841008 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.841957 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.841959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:40.843015 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:40.843024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.843861 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.843865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:40.843957 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:40.843978 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:47:40.844081 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 10:47:40.844151 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":141,"threads_started":1}
I20260927 10:47:40.844197 18376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c53a1d7f9b74290b7071d8834d795c0" candidate_uuid: "490d2f8453b7471cbcb5b45fc1161674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5f4d448414e42da9c8a007328b459ab" is_pre_election: true
I20260927 10:47:40.844511 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:40.839934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34849 (local address 127.0.98.2:44935)
0927 10:47:40.840038 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:40.840042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:40.840077 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:47:40.840118 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:40.840121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:40.840160 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:40.840218 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:40.841139 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.841818 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.843128 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.843710 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.843843 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:40.843902 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:40.844222 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 10:47:40.844360 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 10:47:40.844738 18376 raft_consensus.cc:2454] T 9c53a1d7f9b74290b7071d8834d795c0 P f5f4d448414e42da9c8a007328b459ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 490d2f8453b7471cbcb5b45fc1161674 in term 0.
I20260927 10:47:40.845243 18047 leader_election.cc:302] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 490d2f8453b7471cbcb5b45fc1161674, f5f4d448414e42da9c8a007328b459ab; no voters:
I20260927 10:47:40.845222 18245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c53a1d7f9b74290b7071d8834d795c0" candidate_uuid: "490d2f8453b7471cbcb5b45fc1161674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce5eeb75987d40edace626f6e4ef127a" is_pre_election: true
I20260927 10:47:40.845572 18446 raft_consensus.cc:2781] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:47:40.845597 18245 raft_consensus.cc:2454] T 9c53a1d7f9b74290b7071d8834d795c0 P ce5eeb75987d40edace626f6e4ef127a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 490d2f8453b7471cbcb5b45fc1161674 in term 0.
I20260927 10:47:40.845656 18446 raft_consensus.cc:482] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:47:40.845710 18446 raft_consensus.cc:3037] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:40.846900 18446 raft_consensus.cc:504] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f4d448414e42da9c8a007328b459ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40997 } } peers { permanent_uuid: "ce5eeb75987d40edace626f6e4ef127a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44935 } } peers { permanent_uuid: "490d2f8453b7471cbcb5b45fc1161674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33671 } }
I20260927 10:47:40.847286 18446 leader_election.cc:288] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674 [CANDIDATE]: Term 1 election: Requested vote from peers f5f4d448414e42da9c8a007328b459ab (127.0.98.3:40997), ce5eeb75987d40edace626f6e4ef127a (127.0.98.2:44935)
I20260927 10:47:40.847662 18245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c53a1d7f9b74290b7071d8834d795c0" candidate_uuid: "490d2f8453b7471cbcb5b45fc1161674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce5eeb75987d40edace626f6e4ef127a"
I20260927 10:47:40.847679 18376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c53a1d7f9b74290b7071d8834d795c0" candidate_uuid: "490d2f8453b7471cbcb5b45fc1161674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5f4d448414e42da9c8a007328b459ab"
I20260927 10:47:40.847792 18245 raft_consensus.cc:3037] T 9c53a1d7f9b74290b7071d8834d795c0 P ce5eeb75987d40edace626f6e4ef127a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:40.847796 18376 raft_consensus.cc:3037] T 9c53a1d7f9b74290b7071d8834d795c0 P f5f4d448414e42da9c8a007328b459ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:40.849092 18376 raft_consensus.cc:2454] T 9c53a1d7f9b74290b7071d8834d795c0 P f5f4d448414e42da9c8a007328b459ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 490d2f8453b7471cbcb5b45fc1161674 in term 1.
I20260927 10:47:40.849092 18245 raft_consensus.cc:2454] T 9c53a1d7f9b74290b7071d8834d795c0 P ce5eeb75987d40edace626f6e4ef127a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 490d2f8453b7471cbcb5b45fc1161674 in term 1.
I20260927 10:47:40.849404 18047 leader_election.cc:302] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 490d2f8453b7471cbcb5b45fc1161674, ce5eeb75987d40edace626f6e4ef127a; no voters:
I20260927 10:47:40.849602 18446 raft_consensus.cc:2781] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:40.849844 18446 raft_consensus.cc:686] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674 [term 1 LEADER]: Becoming Leader. State: Replica: 490d2f8453b7471cbcb5b45fc1161674, State: Running, Role: LEADER
I20260927 10:47:40.850175 18446 consensus_queue.cc:237] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f4d448414e42da9c8a007328b459ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40997 } } peers { permanent_uuid: "ce5eeb75987d40edace626f6e4ef127a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44935 } } peers { permanent_uuid: "490d2f8453b7471cbcb5b45fc1161674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33671 } }
I20260927 10:47:40.854112 17972 catalog_manager.cc:5495] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674 reported cstate change: term changed from 0 to 1, leader changed from <none> to 490d2f8453b7471cbcb5b45fc1161674 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "490d2f8453b7471cbcb5b45fc1161674" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f4d448414e42da9c8a007328b459ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce5eeb75987d40edace626f6e4ef127a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "490d2f8453b7471cbcb5b45fc1161674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33671 } health_report { overall_health: HEALTHY } } }
I20260927 10:47:40.875253 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:40.871691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50674 (local address 127.0.98.1:33671)
0927 10:47:40.871824 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:40.871827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:40.871841 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:47:40.871972 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:40.871974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:40.872034 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:40.872123 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:47:40.872391 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.872915 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.873768 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:40.873979 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:40.874066 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:40.874369 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:47:40.874903 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:47:40.874997 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:40.875084 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 10:47:40.875134 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:47:40.881165 18376 raft_consensus.cc:1261] T 9c53a1d7f9b74290b7071d8834d795c0 P f5f4d448414e42da9c8a007328b459ab [term 1 FOLLOWER]: Refusing update from remote peer 490d2f8453b7471cbcb5b45fc1161674: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:47:40.881767 18245 raft_consensus.cc:1261] T 9c53a1d7f9b74290b7071d8834d795c0 P ce5eeb75987d40edace626f6e4ef127a [term 1 FOLLOWER]: Refusing update from remote peer 490d2f8453b7471cbcb5b45fc1161674: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:47:40.882133 18449 consensus_queue.cc:1035] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5f4d448414e42da9c8a007328b459ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:47:40.882704 18446 consensus_queue.cc:1035] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce5eeb75987d40edace626f6e4ef127a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:47:40.889854 18452 mvcc.cc:205] Tried to move back new op lower bound from 7333912825361448960 to 7333912825244884992. Current Snapshot: MvccSnapshot[applied={T|T < 7333912825361448960}]
I20260927 10:47:40.891512 18453 mvcc.cc:205] Tried to move back new op lower bound from 7333912825361448960 to 7333912825244884992. Current Snapshot: MvccSnapshot[applied={T|T < 7333912825361448960}]
I20260927 10:47:40.900112 17972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:47:40.900328 17972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:47:40.903648 17972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f51aef3e88f4ce2898d7621637f3501: Sending DeleteTablet for 3 replicas of tablet 9c53a1d7f9b74290b7071d8834d795c0
I20260927 10:47:40.904625 18225 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c53a1d7f9b74290b7071d8834d795c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:40 UTC) from {username='slave'} at 127.0.0.1:40698
I20260927 10:47:40.904645 18094 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c53a1d7f9b74290b7071d8834d795c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:40 UTC) from {username='slave'} at 127.0.0.1:50670
I20260927 10:47:40.905206 18462 tablet_replica.cc:330] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674: stopping tablet replica
I20260927 10:47:40.905519 18356 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c53a1d7f9b74290b7071d8834d795c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:40 UTC) from {username='slave'} at 127.0.0.1:55042
I20260927 10:47:40.905619 18462 raft_consensus.cc:2229] T 9c53a1d7f9b74290b7071d8834d795c0 P 490d2f8453b7471cbcb5b45fc1161674 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:47:40.905884 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 18032
I20260927 10:47:40.906890 18463 tablet_replica.cc:330] T 9c53a1d7f9b74290b7071d8834d795c0 P ce5eeb75987d40edace626f6e4ef127a: stopping tablet replica
I20260927 10:47:40.907192 18463 raft_consensus.cc:2229] T 9c53a1d7f9b74290b7071d8834d795c0 P ce5eeb75987d40edace626f6e4ef127a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:47:40.907514 18463 raft_consensus.cc:2258] T 9c53a1d7f9b74290b7071d8834d795c0 P ce5eeb75987d40edace626f6e4ef127a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:40.907919 18464 tablet_replica.cc:330] T 9c53a1d7f9b74290b7071d8834d795c0 P f5f4d448414e42da9c8a007328b459ab: stopping tablet replica
I20260927 10:47:40.908244 18464 raft_consensus.cc:2229] T 9c53a1d7f9b74290b7071d8834d795c0 P f5f4d448414e42da9c8a007328b459ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:47:40.908540 18464 raft_consensus.cc:2258] T 9c53a1d7f9b74290b7071d8834d795c0 P f5f4d448414e42da9c8a007328b459ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:40.909180 18463 ts_tablet_manager.cc:1905] T 9c53a1d7f9b74290b7071d8834d795c0 P ce5eeb75987d40edace626f6e4ef127a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:47:40.909956 18464 ts_tablet_manager.cc:1905] T 9c53a1d7f9b74290b7071d8834d795c0 P f5f4d448414e42da9c8a007328b459ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:47:40.912143 18463 ts_tablet_manager.cc:1918] T 9c53a1d7f9b74290b7071d8834d795c0 P ce5eeb75987d40edace626f6e4ef127a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:47:40.912248 18463 log.cc:1196] T 9c53a1d7f9b74290b7071d8834d795c0 P ce5eeb75987d40edace626f6e4ef127a: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-1/wal/wals/9c53a1d7f9b74290b7071d8834d795c0
I20260927 10:47:40.912773 18464 ts_tablet_manager.cc:1918] T 9c53a1d7f9b74290b7071d8834d795c0 P f5f4d448414e42da9c8a007328b459ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:47:40.912876 18464 log.cc:1196] T 9c53a1d7f9b74290b7071d8834d795c0 P f5f4d448414e42da9c8a007328b459ab: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/9c53a1d7f9b74290b7071d8834d795c0
W20260927 10:47:40.912870 17956 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33671 (error 108)
I20260927 10:47:40.912915 18463 ts_tablet_manager.cc:1939] T 9c53a1d7f9b74290b7071d8834d795c0 P ce5eeb75987d40edace626f6e4ef127a: Deleting consensus metadata
I20260927 10:47:40.913180 18464 ts_tablet_manager.cc:1939] T 9c53a1d7f9b74290b7071d8834d795c0 P f5f4d448414e42da9c8a007328b459ab: Deleting consensus metadata
I20260927 10:47:40.913748 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 18163
I20260927 10:47:40.914004 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:40.913691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33671 (local address 127.0.0.1:50686)
0927 10:47:40.913831 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:40.913918 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 10:47:40.915851 17956 catalog_manager.cc:4584] TS 490d2f8453b7471cbcb5b45fc1161674 (127.0.98.1:33671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c53a1d7f9b74290b7071d8834d795c0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33671: connect: Connection refused (error 111)
I20260927 10:47:40.916124 17956 catalog_manager.cc:4841] TS f5f4d448414e42da9c8a007328b459ab (127.0.98.3:40997): tablet 9c53a1d7f9b74290b7071d8834d795c0 (table test-table [id=a81288b69e4949bba7a327da263b6ff0]) successfully deleted
I20260927 10:47:40.920063 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 18294
I20260927 10:47:40.920995 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:40.920231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44935 (local address 127.0.0.1:40712)
0927 10:47:40.920808 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:40.920887 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473}
I20260927 10:47:40.926261 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 17941
2026-09-27T10:47:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1004 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:47:40 dist-test-slave-dzhw krb5kdc[18468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:47:40 dist-test-slave-dzhw krb5kdc[18468](info): set up 2 sockets
Sep 27 10:47:40 dist-test-slave-dzhw krb5kdc[18468](info): commencing operation
krb5kdc: starting...
W20260927 10:47:41.978358 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:47:42 dist-test-slave-dzhw krb5kdc[18468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:47:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:47:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:47:42.101115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43117
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:42.216795 18481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:42.217151 18481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:42.217234 18481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:42.221294 18481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:42.221411 18481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:42.221450 18481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:42.221485 18481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:42.221516 18481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:42.227133 18481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:42.228381 18481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:42.229423 18481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:42.238009 18487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:42.238104 18489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:42.238162 18481 server_base.cc:1034] running on GCE node
W20260927 10:47:42.238364 18486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:42.238986 18481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:42.240069 18481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:42.241250 18481 hybrid_clock.cc:648] HybridClock initialized: now 1790506062241212 us; error 49 us; skew 500 ppm
Sep 27 10:47:42 dist-test-slave-dzhw krb5kdc[18468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:42.251521 18481 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:47:42.252970 18481 webserver.cc:460] Webserver started at http://127.0.98.62:37089/ using document root <none> and password file <none>
I20260927 10:47:42.253787 18481 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:42.253903 18481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:42.254215 18481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:42.257174 18481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "baa51bf06ec846738cec590d69c58283"
format_stamp: "Formatted at 2026-09-27 10:47:42 on dist-test-slave-dzhw"
server_key: "8d36ead166cf627e6893070d6eeb6b83"
server_key_iv: "17d38e244cf1fae9777df5d16f4bd16e"
server_key_version: "encryptionkey@0"
I20260927 10:47:42.257913 18481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "baa51bf06ec846738cec590d69c58283"
format_stamp: "Formatted at 2026-09-27 10:47:42 on dist-test-slave-dzhw"
server_key: "8d36ead166cf627e6893070d6eeb6b83"
server_key_iv: "17d38e244cf1fae9777df5d16f4bd16e"
server_key_version: "encryptionkey@0"
I20260927 10:47:42.263522 18481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260927 10:47:42.266959 18496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:42.268893 18481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:47:42.269063 18481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "baa51bf06ec846738cec590d69c58283"
format_stamp: "Formatted at 2026-09-27 10:47:42 on dist-test-slave-dzhw"
server_key: "8d36ead166cf627e6893070d6eeb6b83"
server_key_iv: "17d38e244cf1fae9777df5d16f4bd16e"
server_key_version: "encryptionkey@0"
I20260927 10:47:42.269233 18481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:42.303135 18481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:42.306589 18481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:42.306754 18481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:42.314061 18481 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43117
I20260927 10:47:42.314082 18548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43117 every 8 connection(s)
I20260927 10:47:42.315203 18481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:47:42.318011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 18481
I20260927 10:47:42.318221 18549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:42.318318 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:47:42.318784 392 external_mini_cluster.cc:1428] Setting key a71cc0fb4ce5485442b92d2744c141a9
I20260927 10:47:42.325071 18549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa51bf06ec846738cec590d69c58283: Bootstrap starting.
I20260927 10:47:42.327504 18549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baa51bf06ec846738cec590d69c58283: Neither blocks nor log segments found. Creating new log.
Sep 27 10:47:42 dist-test-slave-dzhw krb5kdc[18468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:47:42.328457 18549 log.cc:824] T 00000000000000000000000000000000 P baa51bf06ec846738cec590d69c58283: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:42.330966 18549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa51bf06ec846738cec590d69c58283: No bootstrap required, opened a new log
I20260927 10:47:42.334916 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:42.320307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33992 (local address 127.0.98.62:43117)
0927 10:47:42.320797 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:42.320812 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:42.320850 (+ 38us) server_negotiation.cc:407] Connection header received
0927 10:47:42.321688 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:42.321707 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:42.322046 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:42.322468 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:42.323257 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:42.324284 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:42.325185 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:42.325704 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:42.325889 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:42.328795 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:42.328801 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:42.328815 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:42.328846 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:42.331528 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:42.332342 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:42.332347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:42.332352 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:42.332486 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:42.332783 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:42.332785 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:42.332787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:42.333216 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:42.333452 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:42.333739 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 10:47:42.334056 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":141,"threads_started":1}
I20260927 10:47:42.335474 18549 raft_consensus.cc:348] T 00000000000000000000000000000000 P baa51bf06ec846738cec590d69c58283 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa51bf06ec846738cec590d69c58283" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43117 } }
I20260927 10:47:42.335731 18549 raft_consensus.cc:374] T 00000000000000000000000000000000 P baa51bf06ec846738cec590d69c58283 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:42.335804 18549 raft_consensus.cc:729] T 00000000000000000000000000000000 P baa51bf06ec846738cec590d69c58283 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa51bf06ec846738cec590d69c58283, State: Initialized, Role: FOLLOWER
I20260927 10:47:42.336248 18549 consensus_queue.cc:260] T 00000000000000000000000000000000 P baa51bf06ec846738cec590d69c58283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa51bf06ec846738cec590d69c58283" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43117 } }
I20260927 10:47:42.336421 18549 raft_consensus.cc:388] T 00000000000000000000000000000000 P baa51bf06ec846738cec590d69c58283 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:47:42.336499 18549 raft_consensus.cc:482] T 00000000000000000000000000000000 P baa51bf06ec846738cec590d69c58283 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:47:42.336625 18549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baa51bf06ec846738cec590d69c58283 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:42.337597 18549 raft_consensus.cc:504] T 00000000000000000000000000000000 P baa51bf06ec846738cec590d69c58283 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa51bf06ec846738cec590d69c58283" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43117 } }
I20260927 10:47:42.337947 18549 leader_election.cc:302] T 00000000000000000000000000000000 P baa51bf06ec846738cec590d69c58283 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baa51bf06ec846738cec590d69c58283; no voters:
I20260927 10:47:42.338285 18549 leader_election.cc:288] T 00000000000000000000000000000000 P baa51bf06ec846738cec590d69c58283 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:47:42.338459 18554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baa51bf06ec846738cec590d69c58283 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:42.338685 18554 raft_consensus.cc:686] T 00000000000000000000000000000000 P baa51bf06ec846738cec590d69c58283 [term 1 LEADER]: Becoming Leader. State: Replica: baa51bf06ec846738cec590d69c58283, State: Running, Role: LEADER
I20260927 10:47:42.339000 18554 consensus_queue.cc:237] T 00000000000000000000000000000000 P baa51bf06ec846738cec590d69c58283 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa51bf06ec846738cec590d69c58283" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43117 } }
I20260927 10:47:42.339609 18549 sys_catalog.cc:564] T 00000000000000000000000000000000 P baa51bf06ec846738cec590d69c58283 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:47:42.340688 18555 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa51bf06ec846738cec590d69c58283 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "baa51bf06ec846738cec590d69c58283" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa51bf06ec846738cec590d69c58283" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43117 } } }
I20260927 10:47:42.340821 18555 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa51bf06ec846738cec590d69c58283 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:42.341101 18556 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa51bf06ec846738cec590d69c58283 [sys.catalog]: SysCatalogTable state changed. Reason: New leader baa51bf06ec846738cec590d69c58283. Latest consensus state: current_term: 1 leader_uuid: "baa51bf06ec846738cec590d69c58283" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa51bf06ec846738cec590d69c58283" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43117 } } }
I20260927 10:47:42.341190 18556 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa51bf06ec846738cec590d69c58283 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:42.341318 18563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:47:42.344913 18563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:47:42.352342 18563 catalog_manager.cc:1347] Generated new cluster ID: ece840677eae4da5a197c650fead3330
I20260927 10:47:42.352438 18563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:47:42.362303 18563 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:47:42.363644 18563 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:47:42.372051 18563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P baa51bf06ec846738cec590d69c58283: Generated new TSK 0
I20260927 10:47:42.372748 18563 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 10:47:42.405856 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43117
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:42.519992 18575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:42.520314 18575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:42.520360 18575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:42.524582 18575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:42.524652 18575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:42.524711 18575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:47:42.530160 18575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:42.531376 18575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:42.532428 18575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:42.542224 18581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:42.542207 18583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:42.542227 18580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:42.542565 18575 server_base.cc:1034] running on GCE node
I20260927 10:47:42.543022 18575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:42.543668 18575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:42.544855 18575 hybrid_clock.cc:648] HybridClock initialized: now 1790506062544846 us; error 37 us; skew 500 ppm
Sep 27 10:47:42 dist-test-slave-dzhw krb5kdc[18468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:42.552142 18575 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 10:47:42.553130 18575 webserver.cc:460] Webserver started at http://127.0.98.1:33113/ using document root <none> and password file <none>
I20260927 10:47:42.553735 18575 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:42.553800 18575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:42.554028 18575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:42.556200 18575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "70e8acd999ac405da13cc370ba29ba3a"
format_stamp: "Formatted at 2026-09-27 10:47:42 on dist-test-slave-dzhw"
server_key: "d4b882a1b5e1aea59deaa64519346934"
server_key_iv: "867c199cc6732e255c992959fcc96410"
server_key_version: "encryptionkey@0"
I20260927 10:47:42.556767 18575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "70e8acd999ac405da13cc370ba29ba3a"
format_stamp: "Formatted at 2026-09-27 10:47:42 on dist-test-slave-dzhw"
server_key: "d4b882a1b5e1aea59deaa64519346934"
server_key_iv: "867c199cc6732e255c992959fcc96410"
server_key_version: "encryptionkey@0"
I20260927 10:47:42.560585 18575 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:47:42.563175 18590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:42.564546 18575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:47:42.564656 18575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "70e8acd999ac405da13cc370ba29ba3a"
format_stamp: "Formatted at 2026-09-27 10:47:42 on dist-test-slave-dzhw"
server_key: "d4b882a1b5e1aea59deaa64519346934"
server_key_iv: "867c199cc6732e255c992959fcc96410"
server_key_version: "encryptionkey@0"
I20260927 10:47:42.564754 18575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:42.588168 18575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:42.591558 18575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:42.591727 18575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:42.592254 18575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:42.593116 18575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:42.593161 18575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:42.593202 18575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:42.593227 18575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:42.602015 18575 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38465
I20260927 10:47:42.602065 18703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38465 every 8 connection(s)
I20260927 10:47:42.603040 18575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 10:47:42 dist-test-slave-dzhw krb5kdc[18468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:47:42.612984 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 18575
I20260927 10:47:42.613119 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:47:42.613549 392 external_mini_cluster.cc:1428] Setting key fe92a88b9fcb848fb7c08c6f331e431e
I20260927 10:47:42.616952 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:42.604822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36485 (local address 127.0.98.62:43117)
0927 10:47:42.604993 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:42.604998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:42.605801 (+ 803us) server_negotiation.cc:407] Connection header received
0927 10:47:42.607051 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:42.607054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:42.607099 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:42.607176 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:42.608445 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:42.608975 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:42.610122 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:42.610387 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:42.610507 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:42.612980 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:42.612983 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:42.612985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:42.613008 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:42.614775 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:42.615308 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:42.615312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:42.615314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:42.615407 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:42.615791 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:42.615793 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:42.615794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:42.615960 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:42.616034 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:42.616705 (+ 671us) server_negotiation.cc:299] Negotiation successful
0927 10:47:42.616808 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 10:47:42.617889 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:42.605208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43117 (local address 127.0.98.1:36485)
0927 10:47:42.605645 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:42.605688 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:42.606851 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:42.607412 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:42.607419 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:42.607783 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:42.608272 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:42.608284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:42.609100 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:42.609103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:42.609997 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:42.610006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:42.610561 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:42.610564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:42.610697 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:42.611262 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:42.611277 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:42.612771 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:42.614901 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:42.614906 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:42.614919 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:42.615184 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:42.615502 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:42.615505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:42.615507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:42.615697 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:42.616067 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:42.616078 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:42.616439 (+ 361us) client_negotiation.cc:769] Sending connection context
0927 10:47:42.616662 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 10:47:42.617043 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":99,"threads_started":1}
I20260927 10:47:42.619117 18704 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43117
I20260927 10:47:42.619380 18704 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:42.619992 18704 heartbeater.cc:511] Master 127.0.98.62:43117 requested a full tablet report, sending...
I20260927 10:47:42.621651 18511 ts_manager.cc:194] Registered new tserver with Master: 70e8acd999ac405da13cc370ba29ba3a (127.0.98.1:38465)
I20260927 10:47:42.622910 18511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36485
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 10:47:42.642258 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43117
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:42.754098 18709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:42.754446 18709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:42.754530 18709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:42.758702 18709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:42.758788 18709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:42.758888 18709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:47:42.764475 18709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:42.765699 18709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:42.766865 18709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:42.777043 18717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:42.777038 18715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:42.777098 18709 server_base.cc:1034] running on GCE node
W20260927 10:47:42.777043 18714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:42.777786 18709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:42.778577 18709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:42.779794 18709 hybrid_clock.cc:648] HybridClock initialized: now 1790506062779774 us; error 36 us; skew 500 ppm
Sep 27 10:47:42 dist-test-slave-dzhw krb5kdc[18468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:42.782879 18709 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 10:47:42.783815 18709 webserver.cc:460] Webserver started at http://127.0.98.2:46611/ using document root <none> and password file <none>
I20260927 10:47:42.784436 18709 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:42.784533 18709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:42.784766 18709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:42.786911 18709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "c1f7626f79f149f88460ee0193478b4f"
format_stamp: "Formatted at 2026-09-27 10:47:42 on dist-test-slave-dzhw"
server_key: "a2325301a9fbcb4ba84302c5c6a8da6d"
server_key_iv: "89550bb97dceefcb3a5d1da2817373c8"
server_key_version: "encryptionkey@0"
I20260927 10:47:42.787478 18709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c1f7626f79f149f88460ee0193478b4f"
format_stamp: "Formatted at 2026-09-27 10:47:42 on dist-test-slave-dzhw"
server_key: "a2325301a9fbcb4ba84302c5c6a8da6d"
server_key_iv: "89550bb97dceefcb3a5d1da2817373c8"
server_key_version: "encryptionkey@0"
I20260927 10:47:42.791281 18709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 10:47:42.793915 18724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:42.795292 18709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:47:42.795454 18709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "c1f7626f79f149f88460ee0193478b4f"
format_stamp: "Formatted at 2026-09-27 10:47:42 on dist-test-slave-dzhw"
server_key: "a2325301a9fbcb4ba84302c5c6a8da6d"
server_key_iv: "89550bb97dceefcb3a5d1da2817373c8"
server_key_version: "encryptionkey@0"
I20260927 10:47:42.795588 18709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:42.805943 18709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:42.809489 18709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:42.809703 18709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:42.810308 18709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:42.811180 18709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:42.811261 18709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:42.811352 18709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:42.811408 18709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:42.820541 18709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41411
I20260927 10:47:42.820560 18837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41411 every 8 connection(s)
I20260927 10:47:42.821477 18709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:47:42.828900 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 18709
I20260927 10:47:42.829054 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:47:42.829452 392 external_mini_cluster.cc:1428] Setting key 8818792b83d1e161826928efec82f047
Sep 27 10:47:42 dist-test-slave-dzhw krb5kdc[18468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:47:42.835774 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:42.823392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48179 (local address 127.0.98.62:43117)
0927 10:47:42.823537 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:42.823544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:42.824323 (+ 779us) server_negotiation.cc:407] Connection header received
0927 10:47:42.825530 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:42.825532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:42.825580 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:42.825654 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:42.826858 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:42.827373 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:42.828452 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:42.828656 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:42.828734 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:42.831602 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:42.831608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:42.831614 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:42.831650 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:42.833176 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:42.833825 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:42.833828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:42.833829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:42.833900 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:42.834378 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:42.834384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:42.834389 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:42.834705 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:42.834815 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:42.835397 (+ 582us) server_negotiation.cc:299] Negotiation successful
0927 10:47:42.835576 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:47:42.836509 18840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:42.823721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43117 (local address 127.0.98.2:48179)
0927 10:47:42.824153 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:42.824195 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:42.825324 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:42.825807 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:42.825813 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:42.826180 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:42.826735 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:42.826748 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:42.827484 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:42.827486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:42.828342 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:42.828350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:42.828805 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:42.828810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:42.829035 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:42.829778 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:42.829799 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:42.831424 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:42.833298 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:42.833306 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:42.833325 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:42.833721 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:42.834053 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:42.834059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:42.834062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:42.834260 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:42.834828 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:42.834835 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:42.835114 (+ 279us) client_negotiation.cc:769] Sending connection context
0927 10:47:42.835320 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 10:47:42.835610 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":113,"threads_started":1}
I20260927 10:47:42.837739 18838 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43117
I20260927 10:47:42.837996 18838 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:42.838600 18838 heartbeater.cc:511] Master 127.0.98.62:43117 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 10:47:42.839747 18511 ts_manager.cc:194] Registered new tserver with Master: c1f7626f79f149f88460ee0193478b4f (127.0.98.2:41411)
I20260927 10:47:42.840404 18511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 10:47:42.858798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43117
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:42.972220 18843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:42.972548 18843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:42.972641 18843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:42.977099 18843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:42.977195 18843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:42.977291 18843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:47:42.982774 18843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:42.983906 18843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:42.985038 18843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:42.994990 18848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:42.995131 18843 server_base.cc:1034] running on GCE node
W20260927 10:47:42.994998 18849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:42.995009 18851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:42.995648 18843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:42.996332 18843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:42.997527 18843 hybrid_clock.cc:648] HybridClock initialized: now 1790506062997512 us; error 29 us; skew 500 ppm
Sep 27 10:47:42 dist-test-slave-dzhw krb5kdc[18468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:43.000394 18843 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 10:47:43.001366 18843 webserver.cc:460] Webserver started at http://127.0.98.3:38471/ using document root <none> and password file <none>
I20260927 10:47:43.001950 18843 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:43.002043 18843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:43.002280 18843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:43.004518 18843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "2a9dc95ef2f149079e520f3159bcdd91"
format_stamp: "Formatted at 2026-09-27 10:47:42 on dist-test-slave-dzhw"
server_key: "564e8883ce3e6304bf0ed8da6ac35ece"
server_key_iv: "f30f3cee1c832103ccdc5d548db6a5c6"
server_key_version: "encryptionkey@0"
I20260927 10:47:43.005154 18843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2a9dc95ef2f149079e520f3159bcdd91"
format_stamp: "Formatted at 2026-09-27 10:47:42 on dist-test-slave-dzhw"
server_key: "564e8883ce3e6304bf0ed8da6ac35ece"
server_key_iv: "f30f3cee1c832103ccdc5d548db6a5c6"
server_key_version: "encryptionkey@0"
I20260927 10:47:43.009063 18843 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 10:47:43.011668 18858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:43.012949 18843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:47:43.013104 18843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "2a9dc95ef2f149079e520f3159bcdd91"
format_stamp: "Formatted at 2026-09-27 10:47:42 on dist-test-slave-dzhw"
server_key: "564e8883ce3e6304bf0ed8da6ac35ece"
server_key_iv: "f30f3cee1c832103ccdc5d548db6a5c6"
server_key_version: "encryptionkey@0"
I20260927 10:47:43.013237 18843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:43.027192 18843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:43.030504 18843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:43.030712 18843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:43.031298 18843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:43.032133 18843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:43.032203 18843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:43.032275 18843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:43.032327 18843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:43.041407 18843 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44931
I20260927 10:47:43.041500 18971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44931 every 8 connection(s)
I20260927 10:47:43.042459 18843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:47:43.045629 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 18843
I20260927 10:47:43.045776 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:47:43.046087 392 external_mini_cluster.cc:1428] Setting key 7c64a2a9e414492e9524f2f040e974e4
Sep 27 10:47:43 dist-test-slave-dzhw krb5kdc[18468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:47:43.055990 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:43.044349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47513 (local address 127.0.98.62:43117)
0927 10:47:43.044503 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:43.044507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:43.045383 (+ 876us) server_negotiation.cc:407] Connection header received
0927 10:47:43.046804 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:43.046808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:43.046869 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:43.046979 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:43.048164 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.048693 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.049812 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.050046 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.050159 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:43.052594 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:43.052597 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:43.052599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:43.052622 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:43.053900 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:43.054395 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:43.054400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:43.054402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:43.054535 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:43.054887 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:43.054891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:43.054893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:43.055079 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:43.055183 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:43.055690 (+ 507us) server_negotiation.cc:299] Negotiation successful
0927 10:47:43.055810 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 10:47:43.056803 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:43.044716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43117 (local address 127.0.98.3:47513)
0927 10:47:43.045157 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:43.045198 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:43.046601 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:43.047151 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:43.047159 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:43.047512 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:43.048023 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:43.048036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.048820 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.048829 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:43.049685 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:43.049693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.050193 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.050196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:43.050371 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:43.050980 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:43.050995 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:43.052426 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:43.054014 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:43.054020 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:43.054035 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:43.054286 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:43.054632 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:43.054634 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:43.054636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:43.054771 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:43.055192 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:43.055201 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:43.055458 (+ 257us) client_negotiation.cc:769] Sending connection context
0927 10:47:43.055651 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 10:47:43.055989 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":111,"threads_started":1}
I20260927 10:47:43.058207 18972 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43117
I20260927 10:47:43.058537 18972 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:43.059151 18972 heartbeater.cc:511] Master 127.0.98.62:43117 requested a full tablet report, sending...
I20260927 10:47:43.060299 18511 ts_manager.cc:194] Registered new tserver with Master: 2a9dc95ef2f149079e520f3159bcdd91 (127.0.98.3:44931)
I20260927 10:47:43.060942 18511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47513
I20260927 10:47:43.071769 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:47:43.081650 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:43.073789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34008 (local address 127.0.98.62:43117)
0927 10:47:43.073939 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:43.073944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:43.074139 (+ 195us) server_negotiation.cc:407] Connection header received
0927 10:47:43.074205 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:43.074207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:43.074265 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:43.074328 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:43.075172 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.075707 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.076578 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.076777 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.076852 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:43.078516 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:43.078519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:43.078521 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:43.078544 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:43.079938 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:43.080441 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:43.080443 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:43.080445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:43.080516 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:43.080816 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:43.080818 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:43.080819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:43.081021 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:43.081136 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:43.081349 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 10:47:43.081424 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 10:47:43.084971 18511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:47:43.088129 18511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:47:43.102897 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:43.097745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39288 (local address 127.0.98.2:41411)
0927 10:47:43.098477 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:43.098495 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:43.098516 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:47:43.098590 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:43.098596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:43.098792 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:43.098953 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:43.099152 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.099953 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.101437 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.102172 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.102281 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:43.102450 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:43.102629 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 10:47:43.102722 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":111,"threads_started":1}
I20260927 10:47:43.102900 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:43.097312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41411 (local address 127.0.0.1:39288)
0927 10:47:43.098143 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:43.098155 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:43.098227 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:43.098933 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:43.098936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:43.098953 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:43.099038 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:43.099044 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.100080 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.100083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:43.101301 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:43.101311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.102312 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.102317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:43.102443 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:43.102484 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 10:47:43.102597 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 10:47:43.102720 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":95,"thread_start_us":104,"threads_started":1}
I20260927 10:47:43.103288 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:43.096722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57460 (local address 127.0.98.3:44931)
0927 10:47:43.097110 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:43.097122 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:43.097582 (+ 460us) server_negotiation.cc:407] Connection header received
0927 10:47:43.097667 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:43.097674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:43.097876 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:43.098035 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:43.098572 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.099837 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.101655 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.102254 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.102358 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:43.102516 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:43.103039 (+ 523us) server_negotiation.cc:299] Negotiation successful
0927 10:47:43.103141 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":169,"threads_started":1}
I20260927 10:47:43.103281 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:43.096632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38465 (local address 127.0.0.1:37436)
0927 10:47:43.097611 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:43.097625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:43.097695 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:43.098358 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:43.098362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:43.098377 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:43.098535 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:43.098543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.100004 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.100010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:43.101642 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:43.101649 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.102854 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.102858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:43.102945 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:43.102965 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:47:43.103092 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 10:47:43.103154 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":921,"mutex_wait_us":295,"thread_start_us":107,"threads_started":1}
I20260927 10:47:43.103585 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:43.096613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44931 (local address 127.0.0.1:57460)
0927 10:47:43.097257 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:43.097306 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:43.097442 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:43.098226 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:43.098233 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:43.098256 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:43.098433 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:43.098444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.100048 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.100054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:43.101504 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:43.101514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.102525 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.102530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:43.102628 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:43.102656 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 10:47:43.102723 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 10:47:43.102794 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":56,"thread_start_us":80,"threads_started":1}
I20260927 10:47:43.103842 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:43.097407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37436 (local address 127.0.98.1:38465)
0927 10:47:43.097924 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:43.097941 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:43.097964 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:47:43.098037 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:43.098042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:43.098212 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:43.098361 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:43.098704 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.099847 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.101760 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.102567 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.102664 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:43.102796 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:43.103558 (+ 762us) server_negotiation.cc:299] Negotiation successful
0927 10:47:43.103667 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":86,"threads_started":1}
I20260927 10:47:43.106282 18906 tablet_service.cc:1467] Processing CreateTablet for tablet e4e80176af04475dbe1f56fcd5c01f7e (DEFAULT_TABLE table=test-table [id=854f235039254702b0a0325856d9d021]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:43.106376 18638 tablet_service.cc:1467] Processing CreateTablet for tablet e4e80176af04475dbe1f56fcd5c01f7e (DEFAULT_TABLE table=test-table [id=854f235039254702b0a0325856d9d021]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:43.106376 18772 tablet_service.cc:1467] Processing CreateTablet for tablet e4e80176af04475dbe1f56fcd5c01f7e (DEFAULT_TABLE table=test-table [id=854f235039254702b0a0325856d9d021]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:43.107350 18906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e80176af04475dbe1f56fcd5c01f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:43.107404 18638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e80176af04475dbe1f56fcd5c01f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:43.107411 18772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4e80176af04475dbe1f56fcd5c01f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:43.114778 18991 tablet_bootstrap.cc:492] T e4e80176af04475dbe1f56fcd5c01f7e P 70e8acd999ac405da13cc370ba29ba3a: Bootstrap starting.
I20260927 10:47:43.115015 18990 tablet_bootstrap.cc:492] T e4e80176af04475dbe1f56fcd5c01f7e P c1f7626f79f149f88460ee0193478b4f: Bootstrap starting.
I20260927 10:47:43.116760 18991 tablet_bootstrap.cc:654] T e4e80176af04475dbe1f56fcd5c01f7e P 70e8acd999ac405da13cc370ba29ba3a: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:43.117216 18990 tablet_bootstrap.cc:654] T e4e80176af04475dbe1f56fcd5c01f7e P c1f7626f79f149f88460ee0193478b4f: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:43.117868 18991 log.cc:824] T e4e80176af04475dbe1f56fcd5c01f7e P 70e8acd999ac405da13cc370ba29ba3a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:43.117992 18990 log.cc:824] T e4e80176af04475dbe1f56fcd5c01f7e P c1f7626f79f149f88460ee0193478b4f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:43.118759 18989 tablet_bootstrap.cc:492] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91: Bootstrap starting.
I20260927 10:47:43.120007 18990 tablet_bootstrap.cc:492] T e4e80176af04475dbe1f56fcd5c01f7e P c1f7626f79f149f88460ee0193478b4f: No bootstrap required, opened a new log
I20260927 10:47:43.120007 18991 tablet_bootstrap.cc:492] T e4e80176af04475dbe1f56fcd5c01f7e P 70e8acd999ac405da13cc370ba29ba3a: No bootstrap required, opened a new log
I20260927 10:47:43.120201 18991 ts_tablet_manager.cc:1397] T e4e80176af04475dbe1f56fcd5c01f7e P 70e8acd999ac405da13cc370ba29ba3a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:47:43.120201 18990 ts_tablet_manager.cc:1397] T e4e80176af04475dbe1f56fcd5c01f7e P c1f7626f79f149f88460ee0193478b4f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:47:43.121095 18989 tablet_bootstrap.cc:654] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:43.122087 18989 log.cc:824] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:43.123863 18989 tablet_bootstrap.cc:492] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91: No bootstrap required, opened a new log
I20260927 10:47:43.124078 18989 ts_tablet_manager.cc:1397] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 10:47:43.124197 18990 raft_consensus.cc:348] T e4e80176af04475dbe1f56fcd5c01f7e P c1f7626f79f149f88460ee0193478b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9dc95ef2f149079e520f3159bcdd91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44931 } } peers { permanent_uuid: "70e8acd999ac405da13cc370ba29ba3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38465 } } peers { permanent_uuid: "c1f7626f79f149f88460ee0193478b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41411 } }
I20260927 10:47:43.124197 18991 raft_consensus.cc:348] T e4e80176af04475dbe1f56fcd5c01f7e P 70e8acd999ac405da13cc370ba29ba3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9dc95ef2f149079e520f3159bcdd91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44931 } } peers { permanent_uuid: "70e8acd999ac405da13cc370ba29ba3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38465 } } peers { permanent_uuid: "c1f7626f79f149f88460ee0193478b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41411 } }
I20260927 10:47:43.124487 18990 raft_consensus.cc:374] T e4e80176af04475dbe1f56fcd5c01f7e P c1f7626f79f149f88460ee0193478b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:43.124501 18991 raft_consensus.cc:374] T e4e80176af04475dbe1f56fcd5c01f7e P 70e8acd999ac405da13cc370ba29ba3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:43.124552 18991 raft_consensus.cc:729] T e4e80176af04475dbe1f56fcd5c01f7e P 70e8acd999ac405da13cc370ba29ba3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e8acd999ac405da13cc370ba29ba3a, State: Initialized, Role: FOLLOWER
I20260927 10:47:43.124552 18990 raft_consensus.cc:729] T e4e80176af04475dbe1f56fcd5c01f7e P c1f7626f79f149f88460ee0193478b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1f7626f79f149f88460ee0193478b4f, State: Initialized, Role: FOLLOWER
I20260927 10:47:43.125058 18991 consensus_queue.cc:260] T e4e80176af04475dbe1f56fcd5c01f7e P 70e8acd999ac405da13cc370ba29ba3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9dc95ef2f149079e520f3159bcdd91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44931 } } peers { permanent_uuid: "70e8acd999ac405da13cc370ba29ba3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38465 } } peers { permanent_uuid: "c1f7626f79f149f88460ee0193478b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41411 } }
I20260927 10:47:43.125058 18990 consensus_queue.cc:260] T e4e80176af04475dbe1f56fcd5c01f7e P c1f7626f79f149f88460ee0193478b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9dc95ef2f149079e520f3159bcdd91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44931 } } peers { permanent_uuid: "70e8acd999ac405da13cc370ba29ba3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38465 } } peers { permanent_uuid: "c1f7626f79f149f88460ee0193478b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41411 } }
I20260927 10:47:43.125738 18704 heartbeater.cc:503] Master 127.0.98.62:43117 was elected leader, sending a full tablet report...
I20260927 10:47:43.126101 18991 ts_tablet_manager.cc:1428] T e4e80176af04475dbe1f56fcd5c01f7e P 70e8acd999ac405da13cc370ba29ba3a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:47:43.126101 18990 ts_tablet_manager.cc:1428] T e4e80176af04475dbe1f56fcd5c01f7e P c1f7626f79f149f88460ee0193478b4f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:47:43.126458 18838 heartbeater.cc:503] Master 127.0.98.62:43117 was elected leader, sending a full tablet report...
I20260927 10:47:43.127893 18989 raft_consensus.cc:348] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9dc95ef2f149079e520f3159bcdd91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44931 } } peers { permanent_uuid: "70e8acd999ac405da13cc370ba29ba3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38465 } } peers { permanent_uuid: "c1f7626f79f149f88460ee0193478b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41411 } }
I20260927 10:47:43.128183 18989 raft_consensus.cc:374] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:43.128247 18989 raft_consensus.cc:729] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a9dc95ef2f149079e520f3159bcdd91, State: Initialized, Role: FOLLOWER
I20260927 10:47:43.128683 18989 consensus_queue.cc:260] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9dc95ef2f149079e520f3159bcdd91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44931 } } peers { permanent_uuid: "70e8acd999ac405da13cc370ba29ba3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38465 } } peers { permanent_uuid: "c1f7626f79f149f88460ee0193478b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41411 } }
I20260927 10:47:43.129727 18989 ts_tablet_manager.cc:1428] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260927 10:47:43.129873 18972 heartbeater.cc:503] Master 127.0.98.62:43117 was elected leader, sending a full tablet report...
I20260927 10:47:43.175000 18997 raft_consensus.cc:482] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:47:43.175271 18997 raft_consensus.cc:504] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9dc95ef2f149079e520f3159bcdd91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44931 } } peers { permanent_uuid: "70e8acd999ac405da13cc370ba29ba3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38465 } } peers { permanent_uuid: "c1f7626f79f149f88460ee0193478b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41411 } }
I20260927 10:47:43.176708 18997 leader_election.cc:288] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70e8acd999ac405da13cc370ba29ba3a (127.0.98.1:38465), c1f7626f79f149f88460ee0193478b4f (127.0.98.2:41411)
I20260927 10:47:43.180825 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:43.176927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38465 (local address 127.0.98.3:47955)
0927 10:47:43.177092 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:43.177110 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:43.177214 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:43.177543 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:43.177546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:43.177565 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:43.177694 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:43.177702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.178700 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.178703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:43.179594 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:43.179603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.180346 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.180350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:43.180470 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:43.180502 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 10:47:43.180594 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 10:47:43.180666 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":1}
I20260927 10:47:43.180837 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:43.177057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47955 (local address 127.0.98.1:38465)
0927 10:47:43.177324 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:43.177327 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:43.177340 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:47:43.177390 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:43.177393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:43.177429 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:43.177525 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:43.177806 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.178595 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.179735 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.180231 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.180312 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:43.180365 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:43.180627 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 10:47:43.180707 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260927 10:47:43.180959 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:43.177066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53819 (local address 127.0.98.2:41411)
0927 10:47:43.177229 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:43.177234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:43.177638 (+ 404us) server_negotiation.cc:407] Connection header received
0927 10:47:43.177776 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:43.177781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:43.177838 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:43.177896 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:43.178224 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.178822 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.179952 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.180405 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.180506 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:43.180569 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:43.180772 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 10:47:43.180838 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 10:47:43.181017 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:43.176927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41411 (local address 127.0.98.3:53819)
0927 10:47:43.177560 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:43.177573 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:43.177665 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:43.177952 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:43.177956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:43.177974 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:43.178111 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:43.178118 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.178951 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.178954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:43.179830 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:43.179839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.180523 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.180526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:43.180645 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:43.180668 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:47:43.180741 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 10:47:43.180803 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":197,"thread_start_us":182,"threads_started":1}
I20260927 10:47:43.181672 18792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e80176af04475dbe1f56fcd5c01f7e" candidate_uuid: "2a9dc95ef2f149079e520f3159bcdd91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1f7626f79f149f88460ee0193478b4f" is_pre_election: true
I20260927 10:47:43.181675 18658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e80176af04475dbe1f56fcd5c01f7e" candidate_uuid: "2a9dc95ef2f149079e520f3159bcdd91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70e8acd999ac405da13cc370ba29ba3a" is_pre_election: true
I20260927 10:47:43.181998 18792 raft_consensus.cc:2454] T e4e80176af04475dbe1f56fcd5c01f7e P c1f7626f79f149f88460ee0193478b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a9dc95ef2f149079e520f3159bcdd91 in term 0.
I20260927 10:47:43.181998 18658 raft_consensus.cc:2454] T e4e80176af04475dbe1f56fcd5c01f7e P 70e8acd999ac405da13cc370ba29ba3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a9dc95ef2f149079e520f3159bcdd91 in term 0.
I20260927 10:47:43.182542 18860 leader_election.cc:302] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a9dc95ef2f149079e520f3159bcdd91, c1f7626f79f149f88460ee0193478b4f; no voters:
I20260927 10:47:43.182855 18997 raft_consensus.cc:2781] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:47:43.182940 18997 raft_consensus.cc:482] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:47:43.182972 18997 raft_consensus.cc:3037] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:43.184028 18997 raft_consensus.cc:504] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9dc95ef2f149079e520f3159bcdd91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44931 } } peers { permanent_uuid: "70e8acd999ac405da13cc370ba29ba3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38465 } } peers { permanent_uuid: "c1f7626f79f149f88460ee0193478b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41411 } }
I20260927 10:47:43.184422 18997 leader_election.cc:288] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91 [CANDIDATE]: Term 1 election: Requested vote from peers 70e8acd999ac405da13cc370ba29ba3a (127.0.98.1:38465), c1f7626f79f149f88460ee0193478b4f (127.0.98.2:41411)
I20260927 10:47:43.184855 18658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e80176af04475dbe1f56fcd5c01f7e" candidate_uuid: "2a9dc95ef2f149079e520f3159bcdd91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70e8acd999ac405da13cc370ba29ba3a"
I20260927 10:47:43.184877 18792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4e80176af04475dbe1f56fcd5c01f7e" candidate_uuid: "2a9dc95ef2f149079e520f3159bcdd91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1f7626f79f149f88460ee0193478b4f"
I20260927 10:47:43.185001 18658 raft_consensus.cc:3037] T e4e80176af04475dbe1f56fcd5c01f7e P 70e8acd999ac405da13cc370ba29ba3a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:43.185000 18792 raft_consensus.cc:3037] T e4e80176af04475dbe1f56fcd5c01f7e P c1f7626f79f149f88460ee0193478b4f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:43.186362 18792 raft_consensus.cc:2454] T e4e80176af04475dbe1f56fcd5c01f7e P c1f7626f79f149f88460ee0193478b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a9dc95ef2f149079e520f3159bcdd91 in term 1.
I20260927 10:47:43.186371 18658 raft_consensus.cc:2454] T e4e80176af04475dbe1f56fcd5c01f7e P 70e8acd999ac405da13cc370ba29ba3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a9dc95ef2f149079e520f3159bcdd91 in term 1.
I20260927 10:47:43.186724 18860 leader_election.cc:302] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a9dc95ef2f149079e520f3159bcdd91, c1f7626f79f149f88460ee0193478b4f; no voters:
I20260927 10:47:43.186926 18997 raft_consensus.cc:2781] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:43.187160 18997 raft_consensus.cc:686] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91 [term 1 LEADER]: Becoming Leader. State: Replica: 2a9dc95ef2f149079e520f3159bcdd91, State: Running, Role: LEADER
I20260927 10:47:43.187530 18997 consensus_queue.cc:237] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9dc95ef2f149079e520f3159bcdd91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44931 } } peers { permanent_uuid: "70e8acd999ac405da13cc370ba29ba3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38465 } } peers { permanent_uuid: "c1f7626f79f149f88460ee0193478b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41411 } }
I20260927 10:47:43.190806 18511 catalog_manager.cc:5495] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a9dc95ef2f149079e520f3159bcdd91 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2a9dc95ef2f149079e520f3159bcdd91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9dc95ef2f149079e520f3159bcdd91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70e8acd999ac405da13cc370ba29ba3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1f7626f79f149f88460ee0193478b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41411 } health_report { overall_health: UNKNOWN } } }
I20260927 10:47:43.203997 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:43.200151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57472 (local address 127.0.98.3:44931)
0927 10:47:43.200288 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:43.200295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:43.200334 (+ 39us) server_negotiation.cc:407] Connection header received
0927 10:47:43.200505 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:43.200508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:43.200579 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:43.200693 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:47:43.200957 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.201469 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.202350 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.202638 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.202750 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:43.203043 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:47:43.203548 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:47:43.203702 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:43.203796 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 10:47:43.203834 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:47:43.209874 18658 raft_consensus.cc:1261] T e4e80176af04475dbe1f56fcd5c01f7e P 70e8acd999ac405da13cc370ba29ba3a [term 1 FOLLOWER]: Refusing update from remote peer 2a9dc95ef2f149079e520f3159bcdd91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:47:43.209874 18792 raft_consensus.cc:1261] T e4e80176af04475dbe1f56fcd5c01f7e P c1f7626f79f149f88460ee0193478b4f [term 1 FOLLOWER]: Refusing update from remote peer 2a9dc95ef2f149079e520f3159bcdd91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:47:43.210619 18997 consensus_queue.cc:1035] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1f7626f79f149f88460ee0193478b4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:47:43.210831 18999 consensus_queue.cc:1035] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70e8acd999ac405da13cc370ba29ba3a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:47:43.217983 19002 mvcc.cc:205] Tried to move back new op lower bound from 7333912834899951616 to 7333912834819215360. Current Snapshot: MvccSnapshot[applied={T|T < 7333912834899951616}]
I20260927 10:47:43.218602 19003 mvcc.cc:205] Tried to move back new op lower bound from 7333912834899951616 to 7333912834819215360. Current Snapshot: MvccSnapshot[applied={T|T < 7333912834899951616}]
I20260927 10:47:43.227193 18513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:47:43.227407 18513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:47:43.229957 18513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P baa51bf06ec846738cec590d69c58283: Sending DeleteTablet for 3 replicas of tablet e4e80176af04475dbe1f56fcd5c01f7e
I20260927 10:47:43.230926 18638 tablet_service.cc:1514] Processing DeleteTablet for tablet e4e80176af04475dbe1f56fcd5c01f7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37436
I20260927 10:47:43.230944 18772 tablet_service.cc:1514] Processing DeleteTablet for tablet e4e80176af04475dbe1f56fcd5c01f7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39288
I20260927 10:47:43.230944 18906 tablet_service.cc:1514] Processing DeleteTablet for tablet e4e80176af04475dbe1f56fcd5c01f7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57460
I20260927 10:47:43.231451 19013 tablet_replica.cc:330] T e4e80176af04475dbe1f56fcd5c01f7e P c1f7626f79f149f88460ee0193478b4f: stopping tablet replica
I20260927 10:47:43.231633 19012 tablet_replica.cc:330] T e4e80176af04475dbe1f56fcd5c01f7e P 70e8acd999ac405da13cc370ba29ba3a: stopping tablet replica
I20260927 10:47:43.231783 19013 raft_consensus.cc:2229] T e4e80176af04475dbe1f56fcd5c01f7e P c1f7626f79f149f88460ee0193478b4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:47:43.231914 19012 raft_consensus.cc:2229] T e4e80176af04475dbe1f56fcd5c01f7e P 70e8acd999ac405da13cc370ba29ba3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:47:43.232121 19013 raft_consensus.cc:2258] T e4e80176af04475dbe1f56fcd5c01f7e P c1f7626f79f149f88460ee0193478b4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:43.232191 19012 raft_consensus.cc:2258] T e4e80176af04475dbe1f56fcd5c01f7e P 70e8acd999ac405da13cc370ba29ba3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:43.232220 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 18575
I20260927 10:47:43.232522 19011 tablet_replica.cc:330] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91: stopping tablet replica
I20260927 10:47:43.233009 19011 raft_consensus.cc:2229] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:47:43.233398 19011 raft_consensus.cc:2258] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:43.236222 19011 ts_tablet_manager.cc:1905] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:47:43.236517 19013 ts_tablet_manager.cc:1905] T e4e80176af04475dbe1f56fcd5c01f7e P c1f7626f79f149f88460ee0193478b4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:47:43.239037 19013 ts_tablet_manager.cc:1918] T e4e80176af04475dbe1f56fcd5c01f7e P c1f7626f79f149f88460ee0193478b4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:47:43.239037 19011 ts_tablet_manager.cc:1918] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:47:43.239135 19011 log.cc:1196] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/e4e80176af04475dbe1f56fcd5c01f7e
I20260927 10:47:43.239265 19013 log.cc:1196] T e4e80176af04475dbe1f56fcd5c01f7e P c1f7626f79f149f88460ee0193478b4f: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505812134190-392-0/minicluster-data/ts-1/wal/wals/e4e80176af04475dbe1f56fcd5c01f7e
I20260927 10:47:43.239495 19011 ts_tablet_manager.cc:1939] T e4e80176af04475dbe1f56fcd5c01f7e P 2a9dc95ef2f149079e520f3159bcdd91: Deleting consensus metadata
I20260927 10:47:43.239791 19013 ts_tablet_manager.cc:1939] T e4e80176af04475dbe1f56fcd5c01f7e P c1f7626f79f149f88460ee0193478b4f: Deleting consensus metadata
W20260927 10:47:43.240578 18499 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38465 (error 108)
I20260927 10:47:43.240907 18497 catalog_manager.cc:4841] TS 2a9dc95ef2f149079e520f3159bcdd91 (127.0.98.3:44931): tablet e4e80176af04475dbe1f56fcd5c01f7e (table test-table [id=854f235039254702b0a0325856d9d021]) successfully deleted
I20260927 10:47:43.240950 18498 catalog_manager.cc:4841] TS c1f7626f79f149f88460ee0193478b4f (127.0.98.2:41411): tablet e4e80176af04475dbe1f56fcd5c01f7e (table test-table [id=854f235039254702b0a0325856d9d021]) successfully deleted
I20260927 10:47:43.241432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 18709
I20260927 10:47:43.243973 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:43.243613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38465 (local address 127.0.0.1:37440)
0927 10:47:43.243787 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:43.243871 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 10:47:43.244154 18499 catalog_manager.cc:4584] TS 70e8acd999ac405da13cc370ba29ba3a (127.0.98.1:38465): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e4e80176af04475dbe1f56fcd5c01f7e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38465: connect: Connection refused (error 111)
I20260927 10:47:43.247617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 18843
I20260927 10:47:43.253558 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 18481
2026-09-27T10:47:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2327 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T10:47:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:47:43Z Disabled control of system clock
I20260927 10:47:43.284210 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41381
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:47:43.396237 19018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:43.396531 19018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:43.396569 19018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:43.400640 19018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:43.400709 19018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:43.400725 19018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:43.400743 19018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:43.400758 19018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:43.405802 19018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41381
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:43.406912 19018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:43.408227 19018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:43.416689 19024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:43.416895 19018 server_base.cc:1034] running on GCE node
W20260927 10:47:43.416698 19026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:43.416715 19023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:43.417595 19018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:43.418776 19018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:43.419997 19018 hybrid_clock.cc:648] HybridClock initialized: now 1790506063419979 us; error 33 us; skew 500 ppm
I20260927 10:47:43.421988 19018 webserver.cc:460] Webserver started at http://127.0.98.62:42961/ using document root <none> and password file <none>
I20260927 10:47:43.422639 19018 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:43.422732 19018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:43.422971 19018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:43.424990 19018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "015438a9ffd447c1b6e4e8e3efd4190a"
format_stamp: "Formatted at 2026-09-27 10:47:43 on dist-test-slave-dzhw"
server_key: "366f8dfb2fd96d3fdfee93272a842df4"
server_key_iv: "a2c3d22252d5132ddf180f9a842fd8dd"
server_key_version: "encryptionkey@0"
I20260927 10:47:43.425566 19018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "015438a9ffd447c1b6e4e8e3efd4190a"
format_stamp: "Formatted at 2026-09-27 10:47:43 on dist-test-slave-dzhw"
server_key: "366f8dfb2fd96d3fdfee93272a842df4"
server_key_iv: "a2c3d22252d5132ddf180f9a842fd8dd"
server_key_version: "encryptionkey@0"
I20260927 10:47:43.429214 19018 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 10:47:43.431845 19032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:43.433130 19018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:47:43.433243 19018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "015438a9ffd447c1b6e4e8e3efd4190a"
format_stamp: "Formatted at 2026-09-27 10:47:43 on dist-test-slave-dzhw"
server_key: "366f8dfb2fd96d3fdfee93272a842df4"
server_key_iv: "a2c3d22252d5132ddf180f9a842fd8dd"
server_key_version: "encryptionkey@0"
I20260927 10:47:43.433362 19018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:43.448176 19018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:43.448843 19018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:43.448997 19018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:43.455873 19018 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41381
I20260927 10:47:43.455911 19084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41381 every 8 connection(s)
I20260927 10:47:43.456838 19018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:47:43.459957 19085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:43.460427 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 19018
I20260927 10:47:43.460569 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:47:43.461117 392 external_mini_cluster.cc:1428] Setting key 1c45a7d105f34715f5c4b90d00ae07de
I20260927 10:47:43.466737 19085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 015438a9ffd447c1b6e4e8e3efd4190a: Bootstrap starting.
I20260927 10:47:43.469348 19085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 015438a9ffd447c1b6e4e8e3efd4190a: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:43.470170 19085 log.cc:824] T 00000000000000000000000000000000 P 015438a9ffd447c1b6e4e8e3efd4190a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:43.470459 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:43.462545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42584 (local address 127.0.98.62:41381)
0927 10:47:43.463170 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:43.463185 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:43.463221 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:47:43.463878 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:43.463895 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:43.464279 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:43.464573 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:43.464911 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.465896 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.467099 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.467573 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.467692 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:43.468636 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:43.468641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:43.468654 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:47:43.469042 (+ 388us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:43.469098 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:47:43.469118 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:43.469195 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:43.469370 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 10:47:43.469658 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":144,"threads_started":1}
I20260927 10:47:43.472376 19085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 015438a9ffd447c1b6e4e8e3efd4190a: No bootstrap required, opened a new log
I20260927 10:47:43.476728 19085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 015438a9ffd447c1b6e4e8e3efd4190a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015438a9ffd447c1b6e4e8e3efd4190a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41381 } }
I20260927 10:47:43.476975 19085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 015438a9ffd447c1b6e4e8e3efd4190a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:43.477010 19085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 015438a9ffd447c1b6e4e8e3efd4190a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 015438a9ffd447c1b6e4e8e3efd4190a, State: Initialized, Role: FOLLOWER
I20260927 10:47:43.477485 19085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 015438a9ffd447c1b6e4e8e3efd4190a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015438a9ffd447c1b6e4e8e3efd4190a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41381 } }
I20260927 10:47:43.477644 19085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 015438a9ffd447c1b6e4e8e3efd4190a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:47:43.477733 19085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 015438a9ffd447c1b6e4e8e3efd4190a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:47:43.477854 19085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 015438a9ffd447c1b6e4e8e3efd4190a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:43.479104 19085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 015438a9ffd447c1b6e4e8e3efd4190a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015438a9ffd447c1b6e4e8e3efd4190a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41381 } }
I20260927 10:47:43.479508 19085 leader_election.cc:302] T 00000000000000000000000000000000 P 015438a9ffd447c1b6e4e8e3efd4190a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 015438a9ffd447c1b6e4e8e3efd4190a; no voters:
I20260927 10:47:43.479863 19085 leader_election.cc:288] T 00000000000000000000000000000000 P 015438a9ffd447c1b6e4e8e3efd4190a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:47:43.479995 19090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 015438a9ffd447c1b6e4e8e3efd4190a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:43.480258 19090 raft_consensus.cc:686] T 00000000000000000000000000000000 P 015438a9ffd447c1b6e4e8e3efd4190a [term 1 LEADER]: Becoming Leader. State: Replica: 015438a9ffd447c1b6e4e8e3efd4190a, State: Running, Role: LEADER
I20260927 10:47:43.480561 19090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 015438a9ffd447c1b6e4e8e3efd4190a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015438a9ffd447c1b6e4e8e3efd4190a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41381 } }
I20260927 10:47:43.481186 19085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 015438a9ffd447c1b6e4e8e3efd4190a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:47:43.482183 19091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 015438a9ffd447c1b6e4e8e3efd4190a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "015438a9ffd447c1b6e4e8e3efd4190a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015438a9ffd447c1b6e4e8e3efd4190a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41381 } } }
I20260927 10:47:43.482323 19091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 015438a9ffd447c1b6e4e8e3efd4190a [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:43.482597 19092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 015438a9ffd447c1b6e4e8e3efd4190a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 015438a9ffd447c1b6e4e8e3efd4190a. Latest consensus state: current_term: 1 leader_uuid: "015438a9ffd447c1b6e4e8e3efd4190a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015438a9ffd447c1b6e4e8e3efd4190a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41381 } } }
I20260927 10:47:43.482700 19092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 015438a9ffd447c1b6e4e8e3efd4190a [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:43.483538 19099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:47:43.486183 19099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:47:43.491954 19099 catalog_manager.cc:1347] Generated new cluster ID: be346986dc604bcb9251e70bfea692ca
I20260927 10:47:43.492035 19099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:47:43.500816 19099 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:47:43.502038 19099 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:47:43.508863 19099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 015438a9ffd447c1b6e4e8e3efd4190a: Generated new TSK 0
I20260927 10:47:43.509641 19099 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:47:43.514921 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41381
--builtin_ntp_servers=127.0.98.20:35707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:47:43.627306 19109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:43.627594 19109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:43.627633 19109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:43.631880 19109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:43.631937 19109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:43.632028 19109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:47:43.637072 19109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:43.638177 19109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:43.639624 19109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:43.649360 19114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:43.649489 19117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:43.649420 19109 server_base.cc:1034] running on GCE node
W20260927 10:47:43.649561 19115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:43.650072 19109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:43.650777 19109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:43.651970 19109 hybrid_clock.cc:648] HybridClock initialized: now 1790506063651956 us; error 30 us; skew 500 ppm
I20260927 10:47:43.653905 19109 webserver.cc:460] Webserver started at http://127.0.98.1:42863/ using document root <none> and password file <none>
I20260927 10:47:43.654649 19109 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:43.654718 19109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:43.654974 19109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:43.657060 19109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "ea76d1976ee54a78bc9b71ae9185c8e9"
format_stamp: "Formatted at 2026-09-27 10:47:43 on dist-test-slave-dzhw"
server_key: "f852a482cde569b4c399635bc06b3542"
server_key_iv: "064ac3d7d49adbab5ff70d304f7de54f"
server_key_version: "encryptionkey@0"
I20260927 10:47:43.657631 19109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ea76d1976ee54a78bc9b71ae9185c8e9"
format_stamp: "Formatted at 2026-09-27 10:47:43 on dist-test-slave-dzhw"
server_key: "f852a482cde569b4c399635bc06b3542"
server_key_iv: "064ac3d7d49adbab5ff70d304f7de54f"
server_key_version: "encryptionkey@0"
I20260927 10:47:43.661473 19109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 10:47:43.664026 19123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:43.665282 19109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:47:43.665453 19109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "ea76d1976ee54a78bc9b71ae9185c8e9"
format_stamp: "Formatted at 2026-09-27 10:47:43 on dist-test-slave-dzhw"
server_key: "f852a482cde569b4c399635bc06b3542"
server_key_iv: "064ac3d7d49adbab5ff70d304f7de54f"
server_key_version: "encryptionkey@0"
I20260927 10:47:43.665582 19109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:43.691497 19109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:43.692390 19109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:43.692591 19109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:43.693188 19109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:43.694171 19109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:43.694243 19109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:43.694308 19109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:43.694366 19109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:43.703672 19109 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35185
I20260927 10:47:43.703694 19236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35185 every 8 connection(s)
I20260927 10:47:43.704963 19109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:47:43.712445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 19109
I20260927 10:47:43.712590 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:47:43.712884 392 external_mini_cluster.cc:1428] Setting key d2788ea8e7cf439ee9b34971ea411f68
I20260927 10:47:43.714727 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:43.706944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58831 (local address 127.0.98.62:41381)
0927 10:47:43.707077 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:43.707081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:43.708144 (+ 1063us) server_negotiation.cc:407] Connection header received
0927 10:47:43.709555 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:43.709558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:43.709627 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:43.709729 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:43.710682 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.711247 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.712585 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.712849 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.712986 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:43.714221 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:43.714225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:43.714227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:47:43.714240 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:43.714263 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:47:43.714273 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:43.714351 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:43.714560 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 10:47:43.714617 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:47:43.715077 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41381
--builtin_ntp_servers=127.0.98.20:35707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:47:43.716051 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:43.707308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41381 (local address 127.0.98.1:58831)
0927 10:47:43.707981 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:43.708025 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:43.709325 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:43.709906 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:43.709913 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:43.709995 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:43.710545 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:43.710560 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.711375 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.711378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:43.712394 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:43.712406 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.713018 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.713023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:43.713213 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:43.713917 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:47:43.713918 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:43.713960 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:47:43.714027 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:43.714372 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:43.714381 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:43.714387 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:47:43.714544 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 10:47:43.714934 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":126,"threads_started":1}
I20260927 10:47:43.717374 19237 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41381
I20260927 10:47:43.717674 19237 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:43.718328 19237 heartbeater.cc:511] Master 127.0.98.62:41381 requested a full tablet report, sending...
I20260927 10:47:43.720054 19047 ts_manager.cc:194] Registered new tserver with Master: ea76d1976ee54a78bc9b71ae9185c8e9 (127.0.98.1:35185)
I20260927 10:47:43.721202 19047 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58831
W20260927 10:47:43.831188 19240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:43.831472 19240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:43.831513 19240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:43.835656 19240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:43.835727 19240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:43.835784 19240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:47:43.840874 19240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:43.842062 19240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:43.843554 19240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:43.853441 19245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:43.853538 19240 server_base.cc:1034] running on GCE node
W20260927 10:47:43.853456 19248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:43.853437 19246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:43.854075 19240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:43.854856 19240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:43.856051 19240 hybrid_clock.cc:648] HybridClock initialized: now 1790506063856021 us; error 46 us; skew 500 ppm
I20260927 10:47:43.857900 19240 webserver.cc:460] Webserver started at http://127.0.98.2:34269/ using document root <none> and password file <none>
I20260927 10:47:43.858566 19240 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:43.858661 19240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:43.858893 19240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:43.861042 19240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "6966dd7eb6414996b414e1a8c3657377"
format_stamp: "Formatted at 2026-09-27 10:47:43 on dist-test-slave-dzhw"
server_key: "56da5e6facada26bc6b437a3ea38b68a"
server_key_iv: "d14c6845a9fd954ec4d015dbcba4d01d"
server_key_version: "encryptionkey@0"
I20260927 10:47:43.861632 19240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6966dd7eb6414996b414e1a8c3657377"
format_stamp: "Formatted at 2026-09-27 10:47:43 on dist-test-slave-dzhw"
server_key: "56da5e6facada26bc6b437a3ea38b68a"
server_key_iv: "d14c6845a9fd954ec4d015dbcba4d01d"
server_key_version: "encryptionkey@0"
I20260927 10:47:43.865422 19240 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 10:47:43.868010 19254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:43.869304 19240 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:47:43.869444 19240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "6966dd7eb6414996b414e1a8c3657377"
format_stamp: "Formatted at 2026-09-27 10:47:43 on dist-test-slave-dzhw"
server_key: "56da5e6facada26bc6b437a3ea38b68a"
server_key_iv: "d14c6845a9fd954ec4d015dbcba4d01d"
server_key_version: "encryptionkey@0"
I20260927 10:47:43.869576 19240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:43.881244 19240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:43.881968 19240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:43.882174 19240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:43.882850 19240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:43.883843 19240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:43.883914 19240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:43.883973 19240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:43.884037 19240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:43.893071 19240 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33115
I20260927 10:47:43.893095 19367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33115 every 8 connection(s)
I20260927 10:47:43.894066 19240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:47:43.902347 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 19240
I20260927 10:47:43.902544 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:47:43.902863 392 external_mini_cluster.cc:1428] Setting key 7cf0744586878841ec9e1d89c0129ca0
I20260927 10:47:43.904949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41381
--builtin_ntp_servers=127.0.98.20:35707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:47:43.905189 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:43.896036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40817 (local address 127.0.98.62:41381)
0927 10:47:43.896233 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:43.896239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:43.897444 (+ 1205us) server_negotiation.cc:407] Connection header received
0927 10:47:43.899578 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:43.899581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:43.899632 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:43.899707 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:43.900967 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.901492 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.902839 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.903151 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.903277 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:43.904603 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:43.904607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:43.904611 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:47:43.904627 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:43.904650 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:47:43.904660 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:43.904737 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:43.904985 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 10:47:43.905045 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 10:47:43.906101 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:43.896410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41381 (local address 127.0.98.2:40817)
0927 10:47:43.897258 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:43.897318 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:43.899294 (+ 1976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:43.899937 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:43.899949 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:43.900071 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:43.900820 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:43.900838 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.901628 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.901631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:43.902717 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:43.902727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:43.903352 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:43.903358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:43.903568 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:43.904262 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:47:43.904264 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:43.904322 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:47:43.904408 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:43.904800 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:43.904811 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:43.904817 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:47:43.904970 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 10:47:43.905268 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":123,"threads_started":1}
I20260927 10:47:43.907446 19368 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41381
I20260927 10:47:43.907718 19368 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:43.908274 19368 heartbeater.cc:511] Master 127.0.98.62:41381 requested a full tablet report, sending...
I20260927 10:47:43.909407 19047 ts_manager.cc:194] Registered new tserver with Master: 6966dd7eb6414996b414e1a8c3657377 (127.0.98.2:33115)
I20260927 10:47:43.909996 19047 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40817
W20260927 10:47:44.015314 19371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:44.015614 19371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:44.015692 19371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:44.019825 19371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:44.019915 19371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:44.020007 19371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:47:44.024989 19371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:44.026132 19371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:44.027621 19371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:44.037277 19379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:44.037263 19376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:44.037267 19377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:44.037549 19371 server_base.cc:1034] running on GCE node
I20260927 10:47:44.038045 19371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:44.038743 19371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:44.039903 19371 hybrid_clock.cc:648] HybridClock initialized: now 1790506064039888 us; error 27 us; skew 500 ppm
I20260927 10:47:44.041793 19371 webserver.cc:460] Webserver started at http://127.0.98.3:37753/ using document root <none> and password file <none>
I20260927 10:47:44.042367 19371 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:44.042490 19371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:44.042729 19371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:44.044756 19371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "d0cd1c0086d14355bbb375db4406c9e8"
format_stamp: "Formatted at 2026-09-27 10:47:44 on dist-test-slave-dzhw"
server_key: "1acc4b3b2eadb8ab46441e93609093cc"
server_key_iv: "63339bdace1676c9ae679b741a148fce"
server_key_version: "encryptionkey@0"
I20260927 10:47:44.045322 19371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d0cd1c0086d14355bbb375db4406c9e8"
format_stamp: "Formatted at 2026-09-27 10:47:44 on dist-test-slave-dzhw"
server_key: "1acc4b3b2eadb8ab46441e93609093cc"
server_key_iv: "63339bdace1676c9ae679b741a148fce"
server_key_version: "encryptionkey@0"
I20260927 10:47:44.048991 19371 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260927 10:47:44.051456 19385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:44.052661 19371 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:47:44.052804 19371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "d0cd1c0086d14355bbb375db4406c9e8"
format_stamp: "Formatted at 2026-09-27 10:47:44 on dist-test-slave-dzhw"
server_key: "1acc4b3b2eadb8ab46441e93609093cc"
server_key_iv: "63339bdace1676c9ae679b741a148fce"
server_key_version: "encryptionkey@0"
I20260927 10:47:44.052923 19371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:44.075022 19371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:44.075750 19371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:44.075950 19371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:44.076540 19371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:44.077468 19371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:44.077539 19371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:44.077610 19371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:44.077661 19371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:44.087224 19371 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43587
I20260927 10:47:44.087245 19498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43587 every 8 connection(s)
I20260927 10:47:44.088181 19371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:47:44.091956 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 19371
I20260927 10:47:44.092190 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:47:44.092482 392 external_mini_cluster.cc:1428] Setting key 30e66111048792816c6e34b94abab9e6
I20260927 10:47:44.097708 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:44.090116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37673 (local address 127.0.98.62:41381)
0927 10:47:44.090272 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:44.090278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:44.091377 (+ 1099us) server_negotiation.cc:407] Connection header received
0927 10:47:44.092817 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:44.092821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:44.092877 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:44.092955 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:44.094013 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.094665 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.095790 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.096019 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.096127 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:44.097214 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:44.097218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:44.097221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:47:44.097240 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:44.097259 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:47:44.097266 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:44.097343 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:44.097502 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 10:47:44.097563 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:47:44.098690 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:44.090582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41381 (local address 127.0.98.3:37673)
0927 10:47:44.091206 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:44.091246 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:44.092586 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:44.093152 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:44.093158 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:44.093241 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:44.093870 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:44.093892 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.094793 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.094796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:44.095666 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:44.095674 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.096203 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.096211 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:44.096364 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:44.096958 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:47:44.096959 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:44.096999 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:47:44.097063 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:44.097356 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:44.097368 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:44.097379 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 10:47:44.097502 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 10:47:44.097781 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":110,"threads_started":1}
I20260927 10:47:44.099841 19499 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41381
I20260927 10:47:44.100060 19499 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:44.100533 19499 heartbeater.cc:511] Master 127.0.98.62:41381 requested a full tablet report, sending...
I20260927 10:47:44.101485 19047 ts_manager.cc:194] Registered new tserver with Master: d0cd1c0086d14355bbb375db4406c9e8 (127.0.98.3:43587)
I20260927 10:47:44.102016 19047 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:37673
I20260927 10:47:44.106501 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:47:44.112591 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:44.108376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42592 (local address 127.0.98.62:41381)
0927 10:47:44.108572 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:44.108578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:44.108727 (+ 149us) server_negotiation.cc:407] Connection header received
0927 10:47:44.108885 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:44.108888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:44.108937 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:44.109022 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:44.109306 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.109938 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.110839 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.111078 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.111186 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:44.112088 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:44.112093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:44.112101 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:47:44.112113 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:44.112130 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:47:44.112137 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:44.112258 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:44.112394 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 10:47:44.112434 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 10:47:44.115582 19047 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:47:44.118664 19047 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:47:44.133015 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:44.127088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43587 (local address 127.0.0.1:43112)
0927 10:47:44.127502 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:44.127552 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:44.127681 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:44.128954 (+ 1273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:44.128960 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:44.128983 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:44.129086 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:44.129093 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.130281 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.130285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:44.131423 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:44.131434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.132447 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.132450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:44.132574 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:44.132612 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 10:47:44.132725 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 10:47:44.132843 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":53,"thread_start_us":94,"threads_started":1}
I20260927 10:47:44.133261 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:44.127892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43112 (local address 127.0.98.3:43587)
0927 10:47:44.128504 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:44.128516 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:44.128539 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:47:44.128613 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:44.128619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:44.128785 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:44.129280 (+ 495us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:44.129420 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.130123 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.131590 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.132319 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.132445 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:44.132640 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:44.133001 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 10:47:44.133082 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":148,"threads_started":1}
I20260927 10:47:44.133435 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:44.127121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35185 (local address 127.0.0.1:44680)
0927 10:47:44.127684 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:44.127697 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:44.127769 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:44.128683 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:44.128688 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:44.128708 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:44.128857 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:44.128867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.130206 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.130209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:44.131686 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:44.131697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.133074 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.133078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:44.133170 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:44.133189 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 10:47:44.133260 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 10:47:44.133326 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":72,"thread_start_us":90,"threads_started":1}
I20260927 10:47:44.133762 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:44.127344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44680 (local address 127.0.98.1:35185)
0927 10:47:44.128029 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:44.128042 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:44.128067 (+ 25us) server_negotiation.cc:407] Connection header received
0927 10:47:44.128152 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:44.128157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:44.128369 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:44.128517 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:44.128999 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.130083 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.131848 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.132636 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.132750 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:44.132917 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:44.133570 (+ 653us) server_negotiation.cc:299] Negotiation successful
0927 10:47:44.133647 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":92,"threads_started":1}
I20260927 10:47:44.133912 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:44.127838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33115 (local address 127.0.0.1:39612)
0927 10:47:44.128076 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:44.128089 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:44.128166 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:44.129147 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:44.129150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:44.129168 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:44.129247 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:44.129252 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.130596 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.130599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:44.131750 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:44.131760 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.132745 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.132748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:44.132827 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:44.132845 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 10:47:44.132903 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 10:47:44.132957 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260927 10:47:44.134506 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:44.128231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39612 (local address 127.0.98.2:33115)
0927 10:47:44.128629 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:44.128639 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:44.128657 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:47:44.128719 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:44.128723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:44.128889 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:44.129019 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:44.129376 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.130487 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.131874 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.132636 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.132750 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:44.132917 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:44.132972 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 10:47:44.133069 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":118,"threads_started":1}
I20260927 10:47:44.136356 19433 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7deb79f7ba48e19732768a1d6062fa (DEFAULT_TABLE table=test-table [id=a8529e69721f40bbb87e7650c00ac2f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:44.136421 19171 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7deb79f7ba48e19732768a1d6062fa (DEFAULT_TABLE table=test-table [id=a8529e69721f40bbb87e7650c00ac2f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:44.136921 19302 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7deb79f7ba48e19732768a1d6062fa (DEFAULT_TABLE table=test-table [id=a8529e69721f40bbb87e7650c00ac2f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:44.137463 19433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7deb79f7ba48e19732768a1d6062fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:44.137466 19171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7deb79f7ba48e19732768a1d6062fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:44.137768 19302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7deb79f7ba48e19732768a1d6062fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:44.144977 19518 tablet_bootstrap.cc:492] T 2d7deb79f7ba48e19732768a1d6062fa P 6966dd7eb6414996b414e1a8c3657377: Bootstrap starting.
I20260927 10:47:44.145226 19516 tablet_bootstrap.cc:492] T 2d7deb79f7ba48e19732768a1d6062fa P d0cd1c0086d14355bbb375db4406c9e8: Bootstrap starting.
I20260927 10:47:44.145430 19517 tablet_bootstrap.cc:492] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9: Bootstrap starting.
I20260927 10:47:44.147179 19516 tablet_bootstrap.cc:654] T 2d7deb79f7ba48e19732768a1d6062fa P d0cd1c0086d14355bbb375db4406c9e8: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:44.147264 19517 tablet_bootstrap.cc:654] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:44.147264 19518 tablet_bootstrap.cc:654] T 2d7deb79f7ba48e19732768a1d6062fa P 6966dd7eb6414996b414e1a8c3657377: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:44.147956 19516 log.cc:824] T 2d7deb79f7ba48e19732768a1d6062fa P d0cd1c0086d14355bbb375db4406c9e8: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:44.148098 19517 log.cc:824] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:44.148098 19518 log.cc:824] T 2d7deb79f7ba48e19732768a1d6062fa P 6966dd7eb6414996b414e1a8c3657377: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:44.150067 19516 tablet_bootstrap.cc:492] T 2d7deb79f7ba48e19732768a1d6062fa P d0cd1c0086d14355bbb375db4406c9e8: No bootstrap required, opened a new log
I20260927 10:47:44.150067 19518 tablet_bootstrap.cc:492] T 2d7deb79f7ba48e19732768a1d6062fa P 6966dd7eb6414996b414e1a8c3657377: No bootstrap required, opened a new log
I20260927 10:47:44.150079 19517 tablet_bootstrap.cc:492] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9: No bootstrap required, opened a new log
I20260927 10:47:44.150270 19516 ts_tablet_manager.cc:1397] T 2d7deb79f7ba48e19732768a1d6062fa P d0cd1c0086d14355bbb375db4406c9e8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:47:44.150277 19517 ts_tablet_manager.cc:1397] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:47:44.150300 19518 ts_tablet_manager.cc:1397] T 2d7deb79f7ba48e19732768a1d6062fa P 6966dd7eb6414996b414e1a8c3657377: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:47:44.153457 19517 raft_consensus.cc:348] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cd1c0086d14355bbb375db4406c9e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43587 } } peers { permanent_uuid: "ea76d1976ee54a78bc9b71ae9185c8e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35185 } } peers { permanent_uuid: "6966dd7eb6414996b414e1a8c3657377" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33115 } }
I20260927 10:47:44.153719 19517 raft_consensus.cc:374] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:44.153771 19517 raft_consensus.cc:729] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea76d1976ee54a78bc9b71ae9185c8e9, State: Initialized, Role: FOLLOWER
I20260927 10:47:44.154245 19517 consensus_queue.cc:260] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cd1c0086d14355bbb375db4406c9e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43587 } } peers { permanent_uuid: "ea76d1976ee54a78bc9b71ae9185c8e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35185 } } peers { permanent_uuid: "6966dd7eb6414996b414e1a8c3657377" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33115 } }
I20260927 10:47:44.154577 19518 raft_consensus.cc:348] T 2d7deb79f7ba48e19732768a1d6062fa P 6966dd7eb6414996b414e1a8c3657377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cd1c0086d14355bbb375db4406c9e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43587 } } peers { permanent_uuid: "ea76d1976ee54a78bc9b71ae9185c8e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35185 } } peers { permanent_uuid: "6966dd7eb6414996b414e1a8c3657377" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33115 } }
I20260927 10:47:44.154577 19516 raft_consensus.cc:348] T 2d7deb79f7ba48e19732768a1d6062fa P d0cd1c0086d14355bbb375db4406c9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cd1c0086d14355bbb375db4406c9e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43587 } } peers { permanent_uuid: "ea76d1976ee54a78bc9b71ae9185c8e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35185 } } peers { permanent_uuid: "6966dd7eb6414996b414e1a8c3657377" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33115 } }
I20260927 10:47:44.154829 19516 raft_consensus.cc:374] T 2d7deb79f7ba48e19732768a1d6062fa P d0cd1c0086d14355bbb375db4406c9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:44.154829 19518 raft_consensus.cc:374] T 2d7deb79f7ba48e19732768a1d6062fa P 6966dd7eb6414996b414e1a8c3657377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:44.154894 19518 raft_consensus.cc:729] T 2d7deb79f7ba48e19732768a1d6062fa P 6966dd7eb6414996b414e1a8c3657377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6966dd7eb6414996b414e1a8c3657377, State: Initialized, Role: FOLLOWER
I20260927 10:47:44.154894 19516 raft_consensus.cc:729] T 2d7deb79f7ba48e19732768a1d6062fa P d0cd1c0086d14355bbb375db4406c9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cd1c0086d14355bbb375db4406c9e8, State: Initialized, Role: FOLLOWER
I20260927 10:47:44.155100 19237 heartbeater.cc:503] Master 127.0.98.62:41381 was elected leader, sending a full tablet report...
I20260927 10:47:44.155368 19517 ts_tablet_manager.cc:1428] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:47:44.155326 19518 consensus_queue.cc:260] T 2d7deb79f7ba48e19732768a1d6062fa P 6966dd7eb6414996b414e1a8c3657377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cd1c0086d14355bbb375db4406c9e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43587 } } peers { permanent_uuid: "ea76d1976ee54a78bc9b71ae9185c8e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35185 } } peers { permanent_uuid: "6966dd7eb6414996b414e1a8c3657377" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33115 } }
I20260927 10:47:44.155326 19516 consensus_queue.cc:260] T 2d7deb79f7ba48e19732768a1d6062fa P d0cd1c0086d14355bbb375db4406c9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cd1c0086d14355bbb375db4406c9e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43587 } } peers { permanent_uuid: "ea76d1976ee54a78bc9b71ae9185c8e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35185 } } peers { permanent_uuid: "6966dd7eb6414996b414e1a8c3657377" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33115 } }
I20260927 10:47:44.156240 19518 ts_tablet_manager.cc:1428] T 2d7deb79f7ba48e19732768a1d6062fa P 6966dd7eb6414996b414e1a8c3657377: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:47:44.156297 19516 ts_tablet_manager.cc:1428] T 2d7deb79f7ba48e19732768a1d6062fa P d0cd1c0086d14355bbb375db4406c9e8: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:47:44.156577 19499 heartbeater.cc:503] Master 127.0.98.62:41381 was elected leader, sending a full tablet report...
I20260927 10:47:44.157112 19368 heartbeater.cc:503] Master 127.0.98.62:41381 was elected leader, sending a full tablet report...
W20260927 10:47:44.206271 19238 tablet.cc:2369] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:47:44.339207 19500 tablet.cc:2369] T 2d7deb79f7ba48e19732768a1d6062fa P d0cd1c0086d14355bbb375db4406c9e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:47:44.395511 19369 tablet.cc:2369] T 2d7deb79f7ba48e19732768a1d6062fa P 6966dd7eb6414996b414e1a8c3657377: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:47:44.421114 19522 raft_consensus.cc:482] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:47:44.421386 19522 raft_consensus.cc:504] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cd1c0086d14355bbb375db4406c9e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43587 } } peers { permanent_uuid: "ea76d1976ee54a78bc9b71ae9185c8e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35185 } } peers { permanent_uuid: "6966dd7eb6414996b414e1a8c3657377" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33115 } }
I20260927 10:47:44.422572 19522 leader_election.cc:288] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0cd1c0086d14355bbb375db4406c9e8 (127.0.98.3:43587), 6966dd7eb6414996b414e1a8c3657377 (127.0.98.2:33115)
I20260927 10:47:44.426358 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:44.422908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40555 (local address 127.0.98.3:43587)
0927 10:47:44.423042 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:44.423046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:44.423218 (+ 172us) server_negotiation.cc:407] Connection header received
0927 10:47:44.423394 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:44.423396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:44.423436 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:44.423516 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:44.423791 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.424312 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.425397 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.425783 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.425898 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:44.425957 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:44.426149 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 10:47:44.426213 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:47:44.426399 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:44.422945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35857 (local address 127.0.98.2:33115)
0927 10:47:44.423077 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:44.423087 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:44.423373 (+ 286us) server_negotiation.cc:407] Connection header received
0927 10:47:44.423522 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:44.423526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:44.423576 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:44.423662 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:44.423916 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.424403 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.425447 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.425839 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.425957 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:44.426014 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:44.426205 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 10:47:44.426272 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:47:44.426458 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:44.422797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43587 (local address 127.0.98.1:40555)
0927 10:47:44.423124 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:44.423136 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:44.423266 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:44.423538 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:44.423541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:44.423556 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:44.423683 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:44.423689 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.424420 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.424425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:44.425254 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:44.425264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.425916 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.425920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:44.426028 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:44.426046 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 10:47:44.426134 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 10:47:44.426209 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":86,"threads_started":1}
I20260927 10:47:44.426458 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:44.422810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33115 (local address 127.0.98.1:35857)
0927 10:47:44.423284 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:44.423297 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:44.423414 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:44.423673 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:44.423675 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:44.423691 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:44.423811 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:44.423817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.424514 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.424519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:44.425279 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:44.425302 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.425971 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.425975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:44.426073 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:44.426094 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:47:44.426190 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 10:47:44.426261 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":76,"threads_started":1}
I20260927 10:47:44.427122 19453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7deb79f7ba48e19732768a1d6062fa" candidate_uuid: "ea76d1976ee54a78bc9b71ae9185c8e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cd1c0086d14355bbb375db4406c9e8" is_pre_election: true
I20260927 10:47:44.427289 19322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7deb79f7ba48e19732768a1d6062fa" candidate_uuid: "ea76d1976ee54a78bc9b71ae9185c8e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6966dd7eb6414996b414e1a8c3657377" is_pre_election: true
I20260927 10:47:44.427460 19453 raft_consensus.cc:2454] T 2d7deb79f7ba48e19732768a1d6062fa P d0cd1c0086d14355bbb375db4406c9e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea76d1976ee54a78bc9b71ae9185c8e9 in term 0.
I20260927 10:47:44.427515 19322 raft_consensus.cc:2454] T 2d7deb79f7ba48e19732768a1d6062fa P 6966dd7eb6414996b414e1a8c3657377 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea76d1976ee54a78bc9b71ae9185c8e9 in term 0.
I20260927 10:47:44.428018 19126 leader_election.cc:302] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6966dd7eb6414996b414e1a8c3657377, d0cd1c0086d14355bbb375db4406c9e8, ea76d1976ee54a78bc9b71ae9185c8e9; no voters:
I20260927 10:47:44.428299 19522 raft_consensus.cc:2781] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:47:44.428385 19522 raft_consensus.cc:482] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:47:44.428428 19522 raft_consensus.cc:3037] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:44.429461 19522 raft_consensus.cc:504] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cd1c0086d14355bbb375db4406c9e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43587 } } peers { permanent_uuid: "ea76d1976ee54a78bc9b71ae9185c8e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35185 } } peers { permanent_uuid: "6966dd7eb6414996b414e1a8c3657377" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33115 } }
I20260927 10:47:44.429819 19522 leader_election.cc:288] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9 [CANDIDATE]: Term 1 election: Requested vote from peers d0cd1c0086d14355bbb375db4406c9e8 (127.0.98.3:43587), 6966dd7eb6414996b414e1a8c3657377 (127.0.98.2:33115)
I20260927 10:47:44.430159 19453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7deb79f7ba48e19732768a1d6062fa" candidate_uuid: "ea76d1976ee54a78bc9b71ae9185c8e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0cd1c0086d14355bbb375db4406c9e8"
I20260927 10:47:44.430281 19453 raft_consensus.cc:3037] T 2d7deb79f7ba48e19732768a1d6062fa P d0cd1c0086d14355bbb375db4406c9e8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:44.430302 19322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7deb79f7ba48e19732768a1d6062fa" candidate_uuid: "ea76d1976ee54a78bc9b71ae9185c8e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6966dd7eb6414996b414e1a8c3657377"
I20260927 10:47:44.430544 19322 raft_consensus.cc:3037] T 2d7deb79f7ba48e19732768a1d6062fa P 6966dd7eb6414996b414e1a8c3657377 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:44.431347 19453 raft_consensus.cc:2454] T 2d7deb79f7ba48e19732768a1d6062fa P d0cd1c0086d14355bbb375db4406c9e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea76d1976ee54a78bc9b71ae9185c8e9 in term 1.
I20260927 10:47:44.431565 19322 raft_consensus.cc:2454] T 2d7deb79f7ba48e19732768a1d6062fa P 6966dd7eb6414996b414e1a8c3657377 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea76d1976ee54a78bc9b71ae9185c8e9 in term 1.
I20260927 10:47:44.431740 19126 leader_election.cc:302] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0cd1c0086d14355bbb375db4406c9e8, ea76d1976ee54a78bc9b71ae9185c8e9; no voters:
I20260927 10:47:44.431919 19522 raft_consensus.cc:2781] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:44.432202 19522 raft_consensus.cc:686] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9 [term 1 LEADER]: Becoming Leader. State: Replica: ea76d1976ee54a78bc9b71ae9185c8e9, State: Running, Role: LEADER
I20260927 10:47:44.432533 19522 consensus_queue.cc:237] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cd1c0086d14355bbb375db4406c9e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43587 } } peers { permanent_uuid: "ea76d1976ee54a78bc9b71ae9185c8e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35185 } } peers { permanent_uuid: "6966dd7eb6414996b414e1a8c3657377" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33115 } }
I20260927 10:47:44.435814 19049 catalog_manager.cc:5495] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea76d1976ee54a78bc9b71ae9185c8e9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ea76d1976ee54a78bc9b71ae9185c8e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0cd1c0086d14355bbb375db4406c9e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea76d1976ee54a78bc9b71ae9185c8e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6966dd7eb6414996b414e1a8c3657377" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33115 } health_report { overall_health: UNKNOWN } } }
I20260927 10:47:44.498744 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:44.495027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50964 (local address 127.0.98.1:35185)
0927 10:47:44.495151 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:44.495155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:44.495168 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:47:44.495240 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:44.495242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:44.495300 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:44.495401 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:47:44.495718 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.496240 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.497149 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:44.497371 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:44.497481 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:44.497873 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:47:44.498315 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:47:44.498509 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:44.498588 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 10:47:44.498624 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:47:44.505033 19453 raft_consensus.cc:1261] T 2d7deb79f7ba48e19732768a1d6062fa P d0cd1c0086d14355bbb375db4406c9e8 [term 1 FOLLOWER]: Refusing update from remote peer ea76d1976ee54a78bc9b71ae9185c8e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:47:44.505033 19322 raft_consensus.cc:1261] T 2d7deb79f7ba48e19732768a1d6062fa P 6966dd7eb6414996b414e1a8c3657377 [term 1 FOLLOWER]: Refusing update from remote peer ea76d1976ee54a78bc9b71ae9185c8e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:47:44.505745 19527 consensus_queue.cc:1035] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6966dd7eb6414996b414e1a8c3657377" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:47:44.505966 19522 consensus_queue.cc:1035] T 2d7deb79f7ba48e19732768a1d6062fa P ea76d1976ee54a78bc9b71ae9185c8e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0cd1c0086d14355bbb375db4406c9e8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:47:44.513525 19530 mvcc.cc:205] Tried to move back new op lower bound from 7333912840204492800 to 7333912839918370816. Current Snapshot: MvccSnapshot[applied={T|T < 7333912840204492800}]
I20260927 10:47:44.515973 19531 mvcc.cc:205] Tried to move back new op lower bound from 7333912840204492800 to 7333912839918370816. Current Snapshot: MvccSnapshot[applied={T|T < 7333912840204492800}]
I20260927 10:47:44.516285 19532 mvcc.cc:205] Tried to move back new op lower bound from 7333912840204492800 to 7333912839918370816. Current Snapshot: MvccSnapshot[applied={T|T < 7333912840204492800 or (T in {7333912840204492800})}]
I20260927 10:47:44.523418 19047 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:47:44.523607 19047 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:47:44.526090 19047 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 015438a9ffd447c1b6e4e8e3efd4190a: Sending DeleteTablet for 3 replicas of tablet 2d7deb79f7ba48e19732768a1d6062fa
I20260927 10:47:44.526933 19171 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7deb79f7ba48e19732768a1d6062fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:44 UTC) from {username='slave'} at 127.0.0.1:44680
I20260927 10:47:44.526922 19433 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7deb79f7ba48e19732768a1d6062fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:44 UTC) from {username='slave'} at 127.0.0.1:43112
I20260927 10:47:44.527045 19302 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7deb79f7ba48e19732768a1d6062fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:44 UTC) from {username='slave'} at 127.0.0.1:39612
I20260927 10:47:44.527529 19541 tablet_replica.cc:330] T 2d7deb79f7ba48e19732768a1d6062fa P 6966dd7eb6414996b414e1a8c3657377: stopping tablet replica
I20260927 10:47:44.527743 19540 tablet_replica.cc:330] T 2d7deb79f7ba48e19732768a1d6062fa P d0cd1c0086d14355bbb375db4406c9e8: stopping tablet replica
I20260927 10:47:44.527881 19541 raft_consensus.cc:2229] T 2d7deb79f7ba48e19732768a1d6062fa P 6966dd7eb6414996b414e1a8c3657377 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:47:44.528026 19540 raft_consensus.cc:2229] T 2d7deb79f7ba48e19732768a1d6062fa P d0cd1c0086d14355bbb375db4406c9e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:47:44.528215 19541 raft_consensus.cc:2258] T 2d7deb79f7ba48e19732768a1d6062fa P 6966dd7eb6414996b414e1a8c3657377 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:44.528265 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 19109
I20260927 10:47:44.528327 19540 raft_consensus.cc:2258] T 2d7deb79f7ba48e19732768a1d6062fa P d0cd1c0086d14355bbb375db4406c9e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:44.529891 19541 ts_tablet_manager.cc:1905] T 2d7deb79f7ba48e19732768a1d6062fa P 6966dd7eb6414996b414e1a8c3657377: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:47:44.529891 19540 ts_tablet_manager.cc:1905] T 2d7deb79f7ba48e19732768a1d6062fa P d0cd1c0086d14355bbb375db4406c9e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:47:44.532567 19541 ts_tablet_manager.cc:1918] T 2d7deb79f7ba48e19732768a1d6062fa P 6966dd7eb6414996b414e1a8c3657377: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:47:44.532567 19540 ts_tablet_manager.cc:1918] T 2d7deb79f7ba48e19732768a1d6062fa P d0cd1c0086d14355bbb375db4406c9e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:47:44.532675 19541 log.cc:1196] T 2d7deb79f7ba48e19732768a1d6062fa P 6966dd7eb6414996b414e1a8c3657377: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-1/wal/wals/2d7deb79f7ba48e19732768a1d6062fa
I20260927 10:47:44.532675 19540 log.cc:1196] T 2d7deb79f7ba48e19732768a1d6062fa P d0cd1c0086d14355bbb375db4406c9e8: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/2d7deb79f7ba48e19732768a1d6062fa
I20260927 10:47:44.533002 19541 ts_tablet_manager.cc:1939] T 2d7deb79f7ba48e19732768a1d6062fa P 6966dd7eb6414996b414e1a8c3657377: Deleting consensus metadata
I20260927 10:47:44.533002 19540 ts_tablet_manager.cc:1939] T 2d7deb79f7ba48e19732768a1d6062fa P d0cd1c0086d14355bbb375db4406c9e8: Deleting consensus metadata
I20260927 10:47:44.534317 19033 catalog_manager.cc:4841] TS 6966dd7eb6414996b414e1a8c3657377 (127.0.98.2:33115): tablet 2d7deb79f7ba48e19732768a1d6062fa (table test-table [id=a8529e69721f40bbb87e7650c00ac2f1]) successfully deleted
I20260927 10:47:44.534432 19035 catalog_manager.cc:4841] TS d0cd1c0086d14355bbb375db4406c9e8 (127.0.98.3:43587): tablet 2d7deb79f7ba48e19732768a1d6062fa (table test-table [id=a8529e69721f40bbb87e7650c00ac2f1]) successfully deleted
W20260927 10:47:44.535960 19036 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35185 (error 108)
I20260927 10:47:44.536177 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 19240
I20260927 10:47:44.538203 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:44.537945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35185 (local address 127.0.0.1:50978)
0927 10:47:44.538070 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:44.538131 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 10:47:44.538486 19036 catalog_manager.cc:4584] TS ea76d1976ee54a78bc9b71ae9185c8e9 (127.0.98.1:35185): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d7deb79f7ba48e19732768a1d6062fa: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35185: connect: Connection refused (error 111)
I20260927 10:47:44.542007 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 19371
I20260927 10:47:44.547565 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 19018
2026-09-27T10:47:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1292 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:47:44 dist-test-slave-dzhw krb5kdc[19545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:47:44 dist-test-slave-dzhw krb5kdc[19545](info): set up 2 sockets
Sep 27 10:47:44 dist-test-slave-dzhw krb5kdc[19545](info): commencing operation
krb5kdc: starting...
W20260927 10:47:45.578339 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:47:45 dist-test-slave-dzhw krb5kdc[19545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:47:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:47:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:47:45.701033 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36945
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:45.816054 19558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:45.816385 19558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:45.816468 19558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:45.820664 19558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:45.820775 19558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:45.820815 19558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:45.820874 19558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:45.820930 19558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:45.826442 19558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:45.827600 19558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:45.828650 19558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:45.837558 19563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:45.837558 19566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:45.837582 19564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:45.837852 19558 server_base.cc:1034] running on GCE node
I20260927 10:47:45.838546 19558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:45.839690 19558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:45.840886 19558 hybrid_clock.cc:648] HybridClock initialized: now 1790506065840871 us; error 37 us; skew 500 ppm
Sep 27 10:47:45 dist-test-slave-dzhw krb5kdc[19545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:45.843858 19558 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:47:45.844805 19558 webserver.cc:460] Webserver started at http://127.0.98.62:46215/ using document root <none> and password file <none>
I20260927 10:47:45.845466 19558 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:45.845593 19558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:45.845832 19558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:45.848021 19558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "3c1f6da2b39e476da595167ac172570f"
format_stamp: "Formatted at 2026-09-27 10:47:45 on dist-test-slave-dzhw"
server_key: "551f94ed1a89a8c888d5f727a2c072c1"
server_key_iv: "d6dcdf75f0653566e699aa1ace94ff69"
server_key_version: "encryptionkey@0"
I20260927 10:47:45.848583 19558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "3c1f6da2b39e476da595167ac172570f"
format_stamp: "Formatted at 2026-09-27 10:47:45 on dist-test-slave-dzhw"
server_key: "551f94ed1a89a8c888d5f727a2c072c1"
server_key_iv: "d6dcdf75f0653566e699aa1ace94ff69"
server_key_version: "encryptionkey@0"
I20260927 10:47:45.852406 19558 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:47:45.854931 19573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:45.856127 19558 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:47:45.856273 19558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "3c1f6da2b39e476da595167ac172570f"
format_stamp: "Formatted at 2026-09-27 10:47:45 on dist-test-slave-dzhw"
server_key: "551f94ed1a89a8c888d5f727a2c072c1"
server_key_iv: "d6dcdf75f0653566e699aa1ace94ff69"
server_key_version: "encryptionkey@0"
I20260927 10:47:45.856392 19558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:45.866973 19558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:45.870273 19558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:45.870499 19558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:45.876907 19558 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36945
I20260927 10:47:45.876991 19625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36945 every 8 connection(s)
I20260927 10:47:45.878018 19558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:47:45.880949 19626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:45.886615 19626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c1f6da2b39e476da595167ac172570f: Bootstrap starting.
I20260927 10:47:45.887633 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 19558
I20260927 10:47:45.887782 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:47:45.888060 392 external_mini_cluster.cc:1428] Setting key 7f35bec730a382e2a2ffdd0d88ea58eb
I20260927 10:47:45.889418 19626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c1f6da2b39e476da595167ac172570f: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:45.890321 19626 log.cc:824] T 00000000000000000000000000000000 P 3c1f6da2b39e476da595167ac172570f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:45.892534 19626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c1f6da2b39e476da595167ac172570f: No bootstrap required, opened a new log
I20260927 10:47:45.896605 19626 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c1f6da2b39e476da595167ac172570f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c1f6da2b39e476da595167ac172570f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36945 } }
I20260927 10:47:45.896831 19626 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c1f6da2b39e476da595167ac172570f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:45.896893 19626 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c1f6da2b39e476da595167ac172570f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c1f6da2b39e476da595167ac172570f, State: Initialized, Role: FOLLOWER
Sep 27 10:47:45 dist-test-slave-dzhw krb5kdc[19545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:47:45.897389 19626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c1f6da2b39e476da595167ac172570f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c1f6da2b39e476da595167ac172570f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36945 } }
I20260927 10:47:45.897516 19626 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c1f6da2b39e476da595167ac172570f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:47:45.897600 19626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c1f6da2b39e476da595167ac172570f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:47:45.897704 19626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c1f6da2b39e476da595167ac172570f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:45.898917 19626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c1f6da2b39e476da595167ac172570f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c1f6da2b39e476da595167ac172570f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36945 } }
I20260927 10:47:45.899303 19626 leader_election.cc:302] T 00000000000000000000000000000000 P 3c1f6da2b39e476da595167ac172570f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c1f6da2b39e476da595167ac172570f; no voters:
I20260927 10:47:45.899679 19626 leader_election.cc:288] T 00000000000000000000000000000000 P 3c1f6da2b39e476da595167ac172570f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:47:45.899758 19631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c1f6da2b39e476da595167ac172570f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:45.899960 19631 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c1f6da2b39e476da595167ac172570f [term 1 LEADER]: Becoming Leader. State: Replica: 3c1f6da2b39e476da595167ac172570f, State: Running, Role: LEADER
I20260927 10:47:45.900252 19631 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c1f6da2b39e476da595167ac172570f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c1f6da2b39e476da595167ac172570f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36945 } }
I20260927 10:47:45.900998 19626 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c1f6da2b39e476da595167ac172570f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:47:45.901965 19632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c1f6da2b39e476da595167ac172570f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c1f6da2b39e476da595167ac172570f. Latest consensus state: current_term: 1 leader_uuid: "3c1f6da2b39e476da595167ac172570f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c1f6da2b39e476da595167ac172570f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36945 } } }
I20260927 10:47:45.902034 19633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c1f6da2b39e476da595167ac172570f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c1f6da2b39e476da595167ac172570f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c1f6da2b39e476da595167ac172570f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36945 } } }
I20260927 10:47:45.902161 19633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c1f6da2b39e476da595167ac172570f [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:45.902104 19632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c1f6da2b39e476da595167ac172570f [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:45.902607 19640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:47:45.903661 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:45.889208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44838 (local address 127.0.98.62:36945)
0927 10:47:45.889668 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:45.889689 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:45.889733 (+ 44us) server_negotiation.cc:407] Connection header received
0927 10:47:45.890650 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:45.890678 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:45.891034 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:45.891446 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:45.892127 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:45.893582 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:45.894456 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:45.895155 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:45.895347 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:45.897914 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:45.897920 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:45.897933 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:45.897958 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:45.899798 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:45.900320 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:45.900329 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:45.900336 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:45.900475 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:45.900749 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:45.900752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:45.900753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:45.901196 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:45.901404 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:45.902274 (+ 870us) server_negotiation.cc:299] Negotiation successful
0927 10:47:45.902701 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":125,"threads_started":1}
I20260927 10:47:45.905928 19640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:47:45.911305 19640 catalog_manager.cc:1347] Generated new cluster ID: f9bd8f75627147e590fb8a4e527105fb
I20260927 10:47:45.911379 19640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:47:45.936738 19640 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:47:45.937822 19640 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:47:45.951236 19640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c1f6da2b39e476da595167ac172570f: Generated new TSK 0
I20260927 10:47:45.952131 19640 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 10:47:45.989843 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36945
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:46.101766 19652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:46.102115 19652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:46.102191 19652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:46.106382 19652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:46.106500 19652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:46.106606 19652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:47:46.112119 19652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:46.113195 19652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:46.114224 19652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:46.123670 19657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:46.123670 19658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:46.123831 19652 server_base.cc:1034] running on GCE node
W20260927 10:47:46.123984 19660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:46.124388 19652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:46.124948 19652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:46.126113 19652 hybrid_clock.cc:648] HybridClock initialized: now 1790506066126095 us; error 29 us; skew 500 ppm
Sep 27 10:47:46 dist-test-slave-dzhw krb5kdc[19545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:46.128844 19652 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 10:47:46.129617 19652 webserver.cc:460] Webserver started at http://127.0.98.1:33355/ using document root <none> and password file <none>
I20260927 10:47:46.130120 19652 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:46.130175 19652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:46.130329 19652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:46.132275 19652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "40d8eb0cee0e4a4cbc1e1e5e0d9e0764"
format_stamp: "Formatted at 2026-09-27 10:47:46 on dist-test-slave-dzhw"
server_key: "b52a9be1d130d74da7de8d7f4c90cacd"
server_key_iv: "b7995018489403e96e04ec5e91f210a7"
server_key_version: "encryptionkey@0"
I20260927 10:47:46.132748 19652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "40d8eb0cee0e4a4cbc1e1e5e0d9e0764"
format_stamp: "Formatted at 2026-09-27 10:47:46 on dist-test-slave-dzhw"
server_key: "b52a9be1d130d74da7de8d7f4c90cacd"
server_key_iv: "b7995018489403e96e04ec5e91f210a7"
server_key_version: "encryptionkey@0"
I20260927 10:47:46.136207 19652 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:47:46.138535 19667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:46.139747 19652 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:47:46.139878 19652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "40d8eb0cee0e4a4cbc1e1e5e0d9e0764"
format_stamp: "Formatted at 2026-09-27 10:47:46 on dist-test-slave-dzhw"
server_key: "b52a9be1d130d74da7de8d7f4c90cacd"
server_key_iv: "b7995018489403e96e04ec5e91f210a7"
server_key_version: "encryptionkey@0"
I20260927 10:47:46.139994 19652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:46.149356 19652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:46.152089 19652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:46.152285 19652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:46.152832 19652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:46.153621 19652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:46.153693 19652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:46.153760 19652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:46.153808 19652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:46.162919 19652 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46713
I20260927 10:47:46.162940 19780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46713 every 8 connection(s)
I20260927 10:47:46.163753 19652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:47:46.166978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 19652
I20260927 10:47:46.167128 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:47:46.167454 392 external_mini_cluster.cc:1428] Setting key 9f00b1cbfb1afd678df4a75566bae0e7
Sep 27 10:47:46 dist-test-slave-dzhw krb5kdc[19545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 10:47:46.177747 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:46.165563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47205 (local address 127.0.98.62:36945)
0927 10:47:46.165718 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:46.165722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:46.166558 (+ 836us) server_negotiation.cc:407] Connection header received
0927 10:47:46.167517 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:46.167522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:46.167581 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:46.167678 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:46.169135 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.169979 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.171202 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.171443 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.171561 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:46.173971 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:46.173974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:46.173976 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:46.173998 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:46.175546 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:46.176103 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:46.176111 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:46.176112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:46.176186 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:46.176535 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:46.176540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:46.176542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:46.176774 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:46.176880 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:46.177431 (+ 551us) server_negotiation.cc:299] Negotiation successful
0927 10:47:46.177575 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 10:47:46.178841 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:46.165912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.1:47205)
0927 10:47:46.166353 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:46.166392 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:46.167314 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:46.167845 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:46.167854 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:46.168334 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:46.168994 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:46.169009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.170120 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.170123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:46.171068 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:46.171078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.171596 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.171600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:46.171775 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:46.172378 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:46.172393 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:46.173792 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:46.175689 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:46.175695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:46.175711 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:46.175985 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:46.176292 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:46.176295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:46.176297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:46.176437 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:46.176885 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:46.176894 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:46.177150 (+ 256us) client_negotiation.cc:769] Sending connection context
0927 10:47:46.177383 (+ 233us) client_negotiation.cc:240] Negotiation successful
0927 10:47:46.177878 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 10:47:46.182708 19781 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36945
I20260927 10:47:46.182991 19781 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:46.183637 19781 heartbeater.cc:511] Master 127.0.98.62:36945 requested a full tablet report, sending...
I20260927 10:47:46.185482 19590 ts_manager.cc:194] Registered new tserver with Master: 40d8eb0cee0e4a4cbc1e1e5e0d9e0764 (127.0.98.1:46713)
I20260927 10:47:46.187076 19590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 10:47:46.197847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36945
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:46.309688 19786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:46.309957 19786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:46.310000 19786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:46.314181 19786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:46.314245 19786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:46.314311 19786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:47:46.319823 19786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:46.320940 19786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:46.321956 19786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:47:46.331666 19786 server_base.cc:1034] running on GCE node
W20260927 10:47:46.331604 19792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:46.331609 19791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:46.331615 19794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:46.332263 19786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:46.332913 19786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:46.334091 19786 hybrid_clock.cc:648] HybridClock initialized: now 1790506066334078 us; error 31 us; skew 500 ppm
Sep 27 10:47:46 dist-test-slave-dzhw krb5kdc[19545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:46.336761 19786 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 10:47:46.337641 19786 webserver.cc:460] Webserver started at http://127.0.98.2:34757/ using document root <none> and password file <none>
I20260927 10:47:46.338205 19786 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:46.338291 19786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:46.338560 19786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:46.340585 19786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "fd19228f82114092a773f07a2e0232de"
format_stamp: "Formatted at 2026-09-27 10:47:46 on dist-test-slave-dzhw"
server_key: "e572cc264d6dbb0cd937e82e78688c96"
server_key_iv: "15a1d3d2122d68ba1e95405140325eb1"
server_key_version: "encryptionkey@0"
I20260927 10:47:46.341144 19786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fd19228f82114092a773f07a2e0232de"
format_stamp: "Formatted at 2026-09-27 10:47:46 on dist-test-slave-dzhw"
server_key: "e572cc264d6dbb0cd937e82e78688c96"
server_key_iv: "15a1d3d2122d68ba1e95405140325eb1"
server_key_version: "encryptionkey@0"
I20260927 10:47:46.344906 19786 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 10:47:46.347302 19801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:46.348555 19786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260927 10:47:46.348703 19786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "fd19228f82114092a773f07a2e0232de"
format_stamp: "Formatted at 2026-09-27 10:47:46 on dist-test-slave-dzhw"
server_key: "e572cc264d6dbb0cd937e82e78688c96"
server_key_iv: "15a1d3d2122d68ba1e95405140325eb1"
server_key_version: "encryptionkey@0"
I20260927 10:47:46.348816 19786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:46.363780 19786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:46.367128 19786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:46.367336 19786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:46.367924 19786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:46.368767 19786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:46.368842 19786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:46.368928 19786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:46.368975 19786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:46.378103 19786 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45239
I20260927 10:47:46.378126 19914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45239 every 8 connection(s)
I20260927 10:47:46.379092 19786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:47:46.384491 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 19786
I20260927 10:47:46.384625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:47:46.384912 392 external_mini_cluster.cc:1428] Setting key cf58e60c67479126f31dc2045242a6bc
Sep 27 10:47:46 dist-test-slave-dzhw krb5kdc[19545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 10:47:46.393688 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:46.381124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60777 (local address 127.0.98.62:36945)
0927 10:47:46.381332 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:46.381386 (+ 54us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:46.382784 (+ 1398us) server_negotiation.cc:407] Connection header received
0927 10:47:46.384050 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:46.384053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:46.384100 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:46.384175 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:46.385485 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.386208 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.387400 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.387635 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.387743 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:46.390176 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:46.390180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:46.390183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:46.390209 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:46.391648 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:46.392165 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:46.392169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:46.392171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:46.392249 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:46.392609 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:46.392612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:46.392614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:46.392795 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:46.392904 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:46.393399 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 10:47:46.393520 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 10:47:46.394567 19917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:46.381470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.2:60777)
0927 10:47:46.382595 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:46.382656 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:46.383831 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:46.384307 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:46.384314 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:46.384774 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:46.385334 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:46.385348 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.386333 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.386337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:46.387275 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:46.387284 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.387785 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.387789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:46.387964 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:46.388575 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:46.388592 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:46.390012 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:46.391778 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:46.391784 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:46.391800 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:46.392062 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:46.392357 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:46.392360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:46.392362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:46.392501 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:46.392902 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:46.392909 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:46.393166 (+ 257us) client_negotiation.cc:769] Sending connection context
0927 10:47:46.393353 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 10:47:46.393676 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 10:47:46.395925 19915 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36945
I20260927 10:47:46.396175 19915 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:46.396682 19915 heartbeater.cc:511] Master 127.0.98.62:36945 requested a full tablet report, sending...
I20260927 10:47:46.397687 19590 ts_manager.cc:194] Registered new tserver with Master: fd19228f82114092a773f07a2e0232de (127.0.98.2:45239)
I20260927 10:47:46.398263 19590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 10:47:46.411751 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36945
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:46.521770 19920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:46.522049 19920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:46.522092 19920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:46.526158 19920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:46.526232 19920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:46.526291 19920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:47:46.531618 19920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:46.532656 19920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:46.533617 19920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:46.543296 19925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:46.543320 19926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:46.543322 19928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:46.543516 19920 server_base.cc:1034] running on GCE node
I20260927 10:47:46.544015 19920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:46.544718 19920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:46.545902 19920 hybrid_clock.cc:648] HybridClock initialized: now 1790506066545886 us; error 43 us; skew 500 ppm
Sep 27 10:47:46 dist-test-slave-dzhw krb5kdc[19545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:46.548616 19920 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 10:47:46.549532 19920 webserver.cc:460] Webserver started at http://127.0.98.3:45241/ using document root <none> and password file <none>
I20260927 10:47:46.550091 19920 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:46.550181 19920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:46.550400 19920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:46.552456 19920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "877434087a8645a892239036f8fe7b7f"
format_stamp: "Formatted at 2026-09-27 10:47:46 on dist-test-slave-dzhw"
server_key: "5c69bdd4cc07c8fbe9f5a7fa14fe5678"
server_key_iv: "2c2905f44a131d210382c79ebb878e72"
server_key_version: "encryptionkey@0"
I20260927 10:47:46.553050 19920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "877434087a8645a892239036f8fe7b7f"
format_stamp: "Formatted at 2026-09-27 10:47:46 on dist-test-slave-dzhw"
server_key: "5c69bdd4cc07c8fbe9f5a7fa14fe5678"
server_key_iv: "2c2905f44a131d210382c79ebb878e72"
server_key_version: "encryptionkey@0"
I20260927 10:47:46.556675 19920 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:47:46.559034 19935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:46.560205 19920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:47:46.560328 19920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "877434087a8645a892239036f8fe7b7f"
format_stamp: "Formatted at 2026-09-27 10:47:46 on dist-test-slave-dzhw"
server_key: "5c69bdd4cc07c8fbe9f5a7fa14fe5678"
server_key_iv: "2c2905f44a131d210382c79ebb878e72"
server_key_version: "encryptionkey@0"
I20260927 10:47:46.560439 19920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:46.574008 19920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:46.577051 19920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:46.577206 19920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:46.577735 19920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:46.578518 19920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:46.578564 19920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:46.578603 19920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:46.578624 19920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:46.587595 19920 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34271
I20260927 10:47:46.587723 20048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34271 every 8 connection(s)
I20260927 10:47:46.588532 19920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 10:47:46 dist-test-slave-dzhw krb5kdc[19545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:47:46.598083 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 19920
I20260927 10:47:46.598227 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:47:46.598621 392 external_mini_cluster.cc:1428] Setting key 764397fee62de2d1c3df8dd03ed47c52
I20260927 10:47:46.601979 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:46.590317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52681 (local address 127.0.98.62:36945)
0927 10:47:46.590516 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:46.590521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:46.591371 (+ 850us) server_negotiation.cc:407] Connection header received
0927 10:47:46.592186 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:46.592189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:46.592236 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:46.592311 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:46.593463 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.594011 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.595205 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.595422 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.595505 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:46.598037 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:46.598040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:46.598042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:46.598068 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:46.599720 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:46.600279 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:46.600283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:46.600284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:46.600380 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:46.600772 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:46.600775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:46.600777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:46.601118 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:46.601191 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:46.601750 (+ 559us) server_negotiation.cc:299] Negotiation successful
0927 10:47:46.601864 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:47:46.602969 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:46.590756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.3:52681)
0927 10:47:46.591212 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:46.591252 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:46.591999 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:46.592484 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:46.592491 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:46.592841 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:46.593301 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:46.593314 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.594118 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.594120 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:46.595082 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:46.595091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.595547 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.595550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:46.595680 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:46.596272 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:46.596289 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:46.597845 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:46.599854 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:46.599860 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:46.599878 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:46.600183 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:46.600467 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:46.600469 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:46.600470 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:46.600673 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:46.601201 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:46.601206 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:46.601442 (+ 236us) client_negotiation.cc:769] Sending connection context
0927 10:47:46.601694 (+ 252us) client_negotiation.cc:240] Negotiation successful
0927 10:47:46.601964 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":120,"threads_started":1}
I20260927 10:47:46.604113 20049 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36945
I20260927 10:47:46.604359 20049 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:46.604851 20049 heartbeater.cc:511] Master 127.0.98.62:36945 requested a full tablet report, sending...
I20260927 10:47:46.605939 19590 ts_manager.cc:194] Registered new tserver with Master: 877434087a8645a892239036f8fe7b7f (127.0.98.3:34271)
I20260927 10:47:46.606555 19590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52681
I20260927 10:47:46.612377 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:47:46.621331 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:46.614384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44850 (local address 127.0.98.62:36945)
0927 10:47:46.614585 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:46.614588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:46.614672 (+ 84us) server_negotiation.cc:407] Connection header received
0927 10:47:46.614772 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:46.614775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:46.614829 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:46.614896 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:46.615568 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.616130 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.616949 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.617175 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.617267 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:46.618718 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:46.618721 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:46.618723 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:46.618747 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:46.620018 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:46.620376 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:46.620379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:46.620380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:46.620448 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:46.620655 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:46.620656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:46.620658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:46.620856 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:46.620965 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:46.621127 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 10:47:46.621211 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260927 10:47:46.624264 19590 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:47:46.627297 19590 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:47:46.642194 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:46.636569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51632 (local address 127.0.98.1:46713)
0927 10:47:46.637225 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:46.637238 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:46.637262 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:47:46.637375 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:46.637380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:46.637589 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:46.637751 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:46.638216 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.639166 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.640855 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.641499 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.641627 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:46.641796 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:46.641931 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 10:47:46.642024 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":144,"threads_started":1}
I20260927 10:47:46.642191 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:46.635757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46713 (local address 127.0.0.1:51632)
0927 10:47:46.636245 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:46.636290 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:46.636418 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:46.637760 (+ 1342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:46.637766 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:46.637808 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:46.638081 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:46.638085 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.639285 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.639292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:46.640727 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:46.640736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.641625 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.641629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:46.641743 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:46.641781 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 10:47:46.641900 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 10:47:46.642024 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":96,"spinlock_wait_cycles":4864,"thread_start_us":96,"threads_started":1}
I20260927 10:47:46.642760 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:46.636515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34271 (local address 127.0.0.1:36818)
0927 10:47:46.637019 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:46.637034 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:46.637123 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:46.638562 (+ 1439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:46.638565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:46.638584 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:46.638663 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:46.638669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.639948 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.639952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:46.640953 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:46.640964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.642193 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.642197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:46.642421 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:46.642488 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 10:47:46.642560 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 10:47:46.642634 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":94,"thread_start_us":85,"threads_started":1}
I20260927 10:47:46.642897 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:46.636455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53622 (local address 127.0.98.2:45239)
0927 10:47:46.636802 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:46.636814 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:46.637191 (+ 377us) server_negotiation.cc:407] Connection header received
0927 10:47:46.637376 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:46.637380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:46.637589 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:46.637751 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:46.638200 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.639357 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.640989 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.641803 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.641989 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:46.642125 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:46.642625 (+ 500us) server_negotiation.cc:299] Negotiation successful
0927 10:47:46.642718 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":131,"threads_started":1}
I20260927 10:47:46.643185 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:46.636825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36818 (local address 127.0.98.3:34271)
0927 10:47:46.638092 (+ 1267us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:46.638102 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:46.638129 (+ 27us) server_negotiation.cc:407] Connection header received
0927 10:47:46.638221 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:46.638225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:46.638409 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:46.638579 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:46.638771 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.639812 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.641061 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.641803 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.641989 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:46.642125 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:46.642934 (+ 809us) server_negotiation.cc:299] Negotiation successful
0927 10:47:46.643039 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1163,"thread_start_us":94,"threads_started":1}
I20260927 10:47:46.643388 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:46.636753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45239 (local address 127.0.0.1:53622)
0927 10:47:46.637104 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:46.637115 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:46.637252 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:46.637760 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:46.637766 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:46.637809 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:46.638077 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:46.638085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.639488 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.639490 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:46.640862 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:46.640869 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.642193 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.642197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:46.642433 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:46.642484 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 10:47:46.643127 (+ 643us) client_negotiation.cc:240] Negotiation successful
0927 10:47:46.643207 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":170,"spinlock_wait_cycles":9344,"thread_start_us":98,"threads_started":1}
I20260927 10:47:46.645668 19983 tablet_service.cc:1467] Processing CreateTablet for tablet cc34ceb9ae1a4dfa8482b1452037b9cb (DEFAULT_TABLE table=test-table [id=9e774eacd109437ca125c6d5c2a87f04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:46.645728 19715 tablet_service.cc:1467] Processing CreateTablet for tablet cc34ceb9ae1a4dfa8482b1452037b9cb (DEFAULT_TABLE table=test-table [id=9e774eacd109437ca125c6d5c2a87f04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:46.645804 19849 tablet_service.cc:1467] Processing CreateTablet for tablet cc34ceb9ae1a4dfa8482b1452037b9cb (DEFAULT_TABLE table=test-table [id=9e774eacd109437ca125c6d5c2a87f04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:46.646759 19983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc34ceb9ae1a4dfa8482b1452037b9cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:46.646808 19849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc34ceb9ae1a4dfa8482b1452037b9cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:46.646809 19715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc34ceb9ae1a4dfa8482b1452037b9cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:46.653190 20066 tablet_bootstrap.cc:492] T cc34ceb9ae1a4dfa8482b1452037b9cb P 877434087a8645a892239036f8fe7b7f: Bootstrap starting.
I20260927 10:47:46.654026 20067 tablet_bootstrap.cc:492] T cc34ceb9ae1a4dfa8482b1452037b9cb P 40d8eb0cee0e4a4cbc1e1e5e0d9e0764: Bootstrap starting.
I20260927 10:47:46.654735 20068 tablet_bootstrap.cc:492] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de: Bootstrap starting.
I20260927 10:47:46.655458 20066 tablet_bootstrap.cc:654] T cc34ceb9ae1a4dfa8482b1452037b9cb P 877434087a8645a892239036f8fe7b7f: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:46.656129 20067 tablet_bootstrap.cc:654] T cc34ceb9ae1a4dfa8482b1452037b9cb P 40d8eb0cee0e4a4cbc1e1e5e0d9e0764: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:46.656497 20066 log.cc:824] T cc34ceb9ae1a4dfa8482b1452037b9cb P 877434087a8645a892239036f8fe7b7f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:46.656736 20068 tablet_bootstrap.cc:654] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:46.657388 20067 log.cc:824] T cc34ceb9ae1a4dfa8482b1452037b9cb P 40d8eb0cee0e4a4cbc1e1e5e0d9e0764: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:46.657900 20068 log.cc:824] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:46.658540 20066 tablet_bootstrap.cc:492] T cc34ceb9ae1a4dfa8482b1452037b9cb P 877434087a8645a892239036f8fe7b7f: No bootstrap required, opened a new log
I20260927 10:47:46.658735 20066 ts_tablet_manager.cc:1397] T cc34ceb9ae1a4dfa8482b1452037b9cb P 877434087a8645a892239036f8fe7b7f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:47:46.659533 20067 tablet_bootstrap.cc:492] T cc34ceb9ae1a4dfa8482b1452037b9cb P 40d8eb0cee0e4a4cbc1e1e5e0d9e0764: No bootstrap required, opened a new log
I20260927 10:47:46.659720 20067 ts_tablet_manager.cc:1397] T cc34ceb9ae1a4dfa8482b1452037b9cb P 40d8eb0cee0e4a4cbc1e1e5e0d9e0764: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:47:46.659842 20068 tablet_bootstrap.cc:492] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de: No bootstrap required, opened a new log
I20260927 10:47:46.660037 20068 ts_tablet_manager.cc:1397] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:47:46.661957 20066 raft_consensus.cc:348] T cc34ceb9ae1a4dfa8482b1452037b9cb P 877434087a8645a892239036f8fe7b7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d8eb0cee0e4a4cbc1e1e5e0d9e0764" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46713 } } peers { permanent_uuid: "fd19228f82114092a773f07a2e0232de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45239 } } peers { permanent_uuid: "877434087a8645a892239036f8fe7b7f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34271 } }
I20260927 10:47:46.662204 20066 raft_consensus.cc:374] T cc34ceb9ae1a4dfa8482b1452037b9cb P 877434087a8645a892239036f8fe7b7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:46.662256 20066 raft_consensus.cc:729] T cc34ceb9ae1a4dfa8482b1452037b9cb P 877434087a8645a892239036f8fe7b7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 877434087a8645a892239036f8fe7b7f, State: Initialized, Role: FOLLOWER
I20260927 10:47:46.662734 20066 consensus_queue.cc:260] T cc34ceb9ae1a4dfa8482b1452037b9cb P 877434087a8645a892239036f8fe7b7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d8eb0cee0e4a4cbc1e1e5e0d9e0764" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46713 } } peers { permanent_uuid: "fd19228f82114092a773f07a2e0232de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45239 } } peers { permanent_uuid: "877434087a8645a892239036f8fe7b7f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34271 } }
I20260927 10:47:46.663373 20049 heartbeater.cc:503] Master 127.0.98.62:36945 was elected leader, sending a full tablet report...
I20260927 10:47:46.663575 20067 raft_consensus.cc:348] T cc34ceb9ae1a4dfa8482b1452037b9cb P 40d8eb0cee0e4a4cbc1e1e5e0d9e0764 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d8eb0cee0e4a4cbc1e1e5e0d9e0764" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46713 } } peers { permanent_uuid: "fd19228f82114092a773f07a2e0232de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45239 } } peers { permanent_uuid: "877434087a8645a892239036f8fe7b7f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34271 } }
I20260927 10:47:46.663787 20066 ts_tablet_manager.cc:1428] T cc34ceb9ae1a4dfa8482b1452037b9cb P 877434087a8645a892239036f8fe7b7f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:47:46.663812 20067 raft_consensus.cc:374] T cc34ceb9ae1a4dfa8482b1452037b9cb P 40d8eb0cee0e4a4cbc1e1e5e0d9e0764 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:46.663575 20068 raft_consensus.cc:348] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d8eb0cee0e4a4cbc1e1e5e0d9e0764" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46713 } } peers { permanent_uuid: "fd19228f82114092a773f07a2e0232de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45239 } } peers { permanent_uuid: "877434087a8645a892239036f8fe7b7f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34271 } }
I20260927 10:47:46.663863 20067 raft_consensus.cc:729] T cc34ceb9ae1a4dfa8482b1452037b9cb P 40d8eb0cee0e4a4cbc1e1e5e0d9e0764 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40d8eb0cee0e4a4cbc1e1e5e0d9e0764, State: Initialized, Role: FOLLOWER
I20260927 10:47:46.663909 20068 raft_consensus.cc:374] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:46.663962 20068 raft_consensus.cc:729] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd19228f82114092a773f07a2e0232de, State: Initialized, Role: FOLLOWER
I20260927 10:47:46.664316 20068 consensus_queue.cc:260] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d8eb0cee0e4a4cbc1e1e5e0d9e0764" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46713 } } peers { permanent_uuid: "fd19228f82114092a773f07a2e0232de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45239 } } peers { permanent_uuid: "877434087a8645a892239036f8fe7b7f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34271 } }
I20260927 10:47:46.664304 20067 consensus_queue.cc:260] T cc34ceb9ae1a4dfa8482b1452037b9cb P 40d8eb0cee0e4a4cbc1e1e5e0d9e0764 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d8eb0cee0e4a4cbc1e1e5e0d9e0764" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46713 } } peers { permanent_uuid: "fd19228f82114092a773f07a2e0232de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45239 } } peers { permanent_uuid: "877434087a8645a892239036f8fe7b7f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34271 } }
I20260927 10:47:46.665096 20067 ts_tablet_manager.cc:1428] T cc34ceb9ae1a4dfa8482b1452037b9cb P 40d8eb0cee0e4a4cbc1e1e5e0d9e0764: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:47:46.665535 19781 heartbeater.cc:503] Master 127.0.98.62:36945 was elected leader, sending a full tablet report...
I20260927 10:47:46.665545 19915 heartbeater.cc:503] Master 127.0.98.62:36945 was elected leader, sending a full tablet report...
I20260927 10:47:46.665812 20068 ts_tablet_manager.cc:1428] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:47:46.733532 20073 raft_consensus.cc:482] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:47:46.733721 20073 raft_consensus.cc:504] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d8eb0cee0e4a4cbc1e1e5e0d9e0764" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46713 } } peers { permanent_uuid: "fd19228f82114092a773f07a2e0232de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45239 } } peers { permanent_uuid: "877434087a8645a892239036f8fe7b7f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34271 } }
I20260927 10:47:46.734808 20073 leader_election.cc:288] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40d8eb0cee0e4a4cbc1e1e5e0d9e0764 (127.0.98.1:46713), 877434087a8645a892239036f8fe7b7f (127.0.98.3:34271)
I20260927 10:47:46.739267 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:46.735148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38803 (local address 127.0.98.3:34271)
0927 10:47:46.735269 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:46.735273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:46.735286 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:47:46.735382 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:46.735385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:46.735443 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:46.735524 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:46.735814 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.736518 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.738001 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.738644 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.738722 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:46.738778 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:46.739061 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 10:47:46.739130 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:47:46.739250 19917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:46.735055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34271 (local address 127.0.98.2:38803)
0927 10:47:46.735161 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:46.735177 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:46.735269 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:46.735559 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:46.735562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:46.735581 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:46.735698 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:46.735703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.736651 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.736655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:46.737780 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:46.737790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.738752 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.738755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:46.738911 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:46.738937 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 10:47:46.739012 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 10:47:46.739081 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":21}
I20260927 10:47:46.739965 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:46.735270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59837 (local address 127.0.98.1:46713)
0927 10:47:46.735398 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:46.735410 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:46.735835 (+ 425us) server_negotiation.cc:407] Connection header received
0927 10:47:46.735964 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:46.735966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:46.736004 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:46.736057 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:46.736372 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.737131 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.738549 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.739101 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.739193 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:46.739250 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:46.739784 (+ 534us) server_negotiation.cc:299] Negotiation successful
0927 10:47:46.739859 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:47:46.739976 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:46.735055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46713 (local address 127.0.98.2:59837)
0927 10:47:46.735727 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:46.735738 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:46.735864 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:46.736092 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:46.736095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:46.736112 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:46.736226 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:46.736232 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.737272 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.737275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:46.738441 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:46.738451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.739224 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.739228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:46.739652 (+ 424us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:46.739673 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:47:46.739768 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 10:47:46.739833 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":335,"thread_start_us":118,"threads_started":1}
I20260927 10:47:46.740643 20003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc34ceb9ae1a4dfa8482b1452037b9cb" candidate_uuid: "fd19228f82114092a773f07a2e0232de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "877434087a8645a892239036f8fe7b7f" is_pre_election: true
I20260927 10:47:46.740770 19735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc34ceb9ae1a4dfa8482b1452037b9cb" candidate_uuid: "fd19228f82114092a773f07a2e0232de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40d8eb0cee0e4a4cbc1e1e5e0d9e0764" is_pre_election: true
I20260927 10:47:46.741014 20003 raft_consensus.cc:2454] T cc34ceb9ae1a4dfa8482b1452037b9cb P 877434087a8645a892239036f8fe7b7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd19228f82114092a773f07a2e0232de in term 0.
I20260927 10:47:46.741060 19735 raft_consensus.cc:2454] T cc34ceb9ae1a4dfa8482b1452037b9cb P 40d8eb0cee0e4a4cbc1e1e5e0d9e0764 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd19228f82114092a773f07a2e0232de in term 0.
I20260927 10:47:46.741539 19803 leader_election.cc:302] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40d8eb0cee0e4a4cbc1e1e5e0d9e0764, fd19228f82114092a773f07a2e0232de; no voters:
I20260927 10:47:46.741833 20073 raft_consensus.cc:2781] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:47:46.741925 20073 raft_consensus.cc:482] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:47:46.741976 20073 raft_consensus.cc:3037] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:46.743152 20073 raft_consensus.cc:504] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d8eb0cee0e4a4cbc1e1e5e0d9e0764" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46713 } } peers { permanent_uuid: "fd19228f82114092a773f07a2e0232de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45239 } } peers { permanent_uuid: "877434087a8645a892239036f8fe7b7f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34271 } }
I20260927 10:47:46.743569 20073 leader_election.cc:288] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de [CANDIDATE]: Term 1 election: Requested vote from peers 40d8eb0cee0e4a4cbc1e1e5e0d9e0764 (127.0.98.1:46713), 877434087a8645a892239036f8fe7b7f (127.0.98.3:34271)
I20260927 10:47:46.743952 19735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc34ceb9ae1a4dfa8482b1452037b9cb" candidate_uuid: "fd19228f82114092a773f07a2e0232de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40d8eb0cee0e4a4cbc1e1e5e0d9e0764"
I20260927 10:47:46.744060 20003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc34ceb9ae1a4dfa8482b1452037b9cb" candidate_uuid: "fd19228f82114092a773f07a2e0232de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "877434087a8645a892239036f8fe7b7f"
I20260927 10:47:46.744086 19735 raft_consensus.cc:3037] T cc34ceb9ae1a4dfa8482b1452037b9cb P 40d8eb0cee0e4a4cbc1e1e5e0d9e0764 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:46.744199 20003 raft_consensus.cc:3037] T cc34ceb9ae1a4dfa8482b1452037b9cb P 877434087a8645a892239036f8fe7b7f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:46.745231 19735 raft_consensus.cc:2454] T cc34ceb9ae1a4dfa8482b1452037b9cb P 40d8eb0cee0e4a4cbc1e1e5e0d9e0764 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd19228f82114092a773f07a2e0232de in term 1.
I20260927 10:47:46.745558 19803 leader_election.cc:302] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40d8eb0cee0e4a4cbc1e1e5e0d9e0764, fd19228f82114092a773f07a2e0232de; no voters:
I20260927 10:47:46.745731 20073 raft_consensus.cc:2781] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:46.745723 20003 raft_consensus.cc:2454] T cc34ceb9ae1a4dfa8482b1452037b9cb P 877434087a8645a892239036f8fe7b7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd19228f82114092a773f07a2e0232de in term 1.
I20260927 10:47:46.745947 20073 raft_consensus.cc:686] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de [term 1 LEADER]: Becoming Leader. State: Replica: fd19228f82114092a773f07a2e0232de, State: Running, Role: LEADER
I20260927 10:47:46.746322 20073 consensus_queue.cc:237] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d8eb0cee0e4a4cbc1e1e5e0d9e0764" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46713 } } peers { permanent_uuid: "fd19228f82114092a773f07a2e0232de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45239 } } peers { permanent_uuid: "877434087a8645a892239036f8fe7b7f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34271 } }
I20260927 10:47:46.749603 19587 catalog_manager.cc:5495] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de reported cstate change: term changed from 0 to 1, leader changed from <none> to fd19228f82114092a773f07a2e0232de (127.0.98.2). New cstate: current_term: 1 leader_uuid: "fd19228f82114092a773f07a2e0232de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d8eb0cee0e4a4cbc1e1e5e0d9e0764" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd19228f82114092a773f07a2e0232de" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "877434087a8645a892239036f8fe7b7f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34271 } health_report { overall_health: UNKNOWN } } }
I20260927 10:47:46.764045 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:46.760490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53632 (local address 127.0.98.2:45239)
0927 10:47:46.760625 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:46.760629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:46.760644 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:47:46.760783 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:46.760786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:46.760839 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:46.760949 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:47:46.761183 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.761697 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.762583 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.762811 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.762919 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:46.763222 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:47:46.763639 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:47:46.763767 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:46.763826 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 10:47:46.763881 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:47:46.769984 19735 raft_consensus.cc:1261] T cc34ceb9ae1a4dfa8482b1452037b9cb P 40d8eb0cee0e4a4cbc1e1e5e0d9e0764 [term 1 FOLLOWER]: Refusing update from remote peer fd19228f82114092a773f07a2e0232de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:47:46.769984 20003 raft_consensus.cc:1261] T cc34ceb9ae1a4dfa8482b1452037b9cb P 877434087a8645a892239036f8fe7b7f [term 1 FOLLOWER]: Refusing update from remote peer fd19228f82114092a773f07a2e0232de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:47:46.770793 20076 consensus_queue.cc:1035] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de [LEADER]: Connected to new peer: Peer: permanent_uuid: "877434087a8645a892239036f8fe7b7f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:47:46.771023 20073 consensus_queue.cc:1035] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de [LEADER]: Connected to new peer: Peer: permanent_uuid: "40d8eb0cee0e4a4cbc1e1e5e0d9e0764" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:47:46.780750 20081 mvcc.cc:205] Tried to move back new op lower bound from 7333912849481969664 to 7333912849395642368. Current Snapshot: MvccSnapshot[applied={T|T < 7333912849481969664}]
I20260927 10:47:46.785611 19587 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:47:46.785786 19587 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:47:46.788254 19587 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c1f6da2b39e476da595167ac172570f: Sending DeleteTablet for 3 replicas of tablet cc34ceb9ae1a4dfa8482b1452037b9cb
I20260927 10:47:46.789069 19715 tablet_service.cc:1514] Processing DeleteTablet for tablet cc34ceb9ae1a4dfa8482b1452037b9cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51632
I20260927 10:47:46.789101 19849 tablet_service.cc:1514] Processing DeleteTablet for tablet cc34ceb9ae1a4dfa8482b1452037b9cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53622
I20260927 10:47:46.789115 19983 tablet_service.cc:1514] Processing DeleteTablet for tablet cc34ceb9ae1a4dfa8482b1452037b9cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36818
I20260927 10:47:46.789695 20088 tablet_replica.cc:330] T cc34ceb9ae1a4dfa8482b1452037b9cb P 40d8eb0cee0e4a4cbc1e1e5e0d9e0764: stopping tablet replica
I20260927 10:47:46.789695 20089 tablet_replica.cc:330] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de: stopping tablet replica
I20260927 10:47:46.790045 20089 raft_consensus.cc:2229] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:47:46.790103 20088 raft_consensus.cc:2229] T cc34ceb9ae1a4dfa8482b1452037b9cb P 40d8eb0cee0e4a4cbc1e1e5e0d9e0764 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:47:46.790330 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 19652
I20260927 10:47:46.792322 20089 raft_consensus.cc:2258] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:46.792748 20090 tablet_replica.cc:330] T cc34ceb9ae1a4dfa8482b1452037b9cb P 877434087a8645a892239036f8fe7b7f: stopping tablet replica
I20260927 10:47:46.793093 20090 raft_consensus.cc:2229] T cc34ceb9ae1a4dfa8482b1452037b9cb P 877434087a8645a892239036f8fe7b7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:47:46.793404 20090 raft_consensus.cc:2258] T cc34ceb9ae1a4dfa8482b1452037b9cb P 877434087a8645a892239036f8fe7b7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:46.794857 20089 ts_tablet_manager.cc:1905] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:47:46.795682 20090 ts_tablet_manager.cc:1905] T cc34ceb9ae1a4dfa8482b1452037b9cb P 877434087a8645a892239036f8fe7b7f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 10:47:46.797833 19575 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46713 (error 108)
I20260927 10:47:46.797921 20089 ts_tablet_manager.cc:1918] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:47:46.798022 20089 log.cc:1196] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-1/wal/wals/cc34ceb9ae1a4dfa8482b1452037b9cb
I20260927 10:47:46.798358 20089 ts_tablet_manager.cc:1939] T cc34ceb9ae1a4dfa8482b1452037b9cb P fd19228f82114092a773f07a2e0232de: Deleting consensus metadata
I20260927 10:47:46.798734 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 19786
I20260927 10:47:46.798772 20090 ts_tablet_manager.cc:1918] T cc34ceb9ae1a4dfa8482b1452037b9cb P 877434087a8645a892239036f8fe7b7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:47:46.799835 20090 log.cc:1196] T cc34ceb9ae1a4dfa8482b1452037b9cb P 877434087a8645a892239036f8fe7b7f: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/cc34ceb9ae1a4dfa8482b1452037b9cb
I20260927 10:47:46.800160 20090 ts_tablet_manager.cc:1939] T cc34ceb9ae1a4dfa8482b1452037b9cb P 877434087a8645a892239036f8fe7b7f: Deleting consensus metadata
I20260927 10:47:46.801151 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:46.800822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46713 (local address 127.0.0.1:51640)
0927 10:47:46.800985 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:46.801063 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 10:47:46.801388 19577 catalog_manager.cc:4841] TS 877434087a8645a892239036f8fe7b7f (127.0.98.3:34271): tablet cc34ceb9ae1a4dfa8482b1452037b9cb (table test-table [id=9e774eacd109437ca125c6d5c2a87f04]) successfully deleted
W20260927 10:47:46.801390 19575 catalog_manager.cc:4584] TS 40d8eb0cee0e4a4cbc1e1e5e0d9e0764 (127.0.98.1:46713): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc34ceb9ae1a4dfa8482b1452037b9cb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46713: connect: Connection refused (error 111)
I20260927 10:47:46.804529 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 19920
I20260927 10:47:46.805361 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:46.804734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45239 (local address 127.0.0.1:53636)
0927 10:47:46.805178 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:46.805240 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343}
I20260927 10:47:46.810030 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 19558
2026-09-27T10:47:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2263 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T10:47:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:47:46Z Disabled control of system clock
I20260927 10:47:46.840306 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32869
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:47:46.950098 20095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:46.950382 20095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:46.950505 20095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:46.954526 20095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:46.954608 20095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:46.954640 20095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:46.954660 20095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:46.954676 20095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:46.959985 20095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32869
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:46.961102 20095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:46.962471 20095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:46.971062 20101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:46.971143 20103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:46.971062 20100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:46.971298 20095 server_base.cc:1034] running on GCE node
I20260927 10:47:46.972007 20095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:46.973172 20095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:46.974363 20095 hybrid_clock.cc:648] HybridClock initialized: now 1790506066974330 us; error 47 us; skew 500 ppm
I20260927 10:47:46.976332 20095 webserver.cc:460] Webserver started at http://127.0.98.62:34977/ using document root <none> and password file <none>
I20260927 10:47:46.976954 20095 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:46.977046 20095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:46.977269 20095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:46.979399 20095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "3c2864e60e6e44ecbc10d219ab31008e"
format_stamp: "Formatted at 2026-09-27 10:47:46 on dist-test-slave-dzhw"
server_key: "6582a7c56c43120a1cd599bc731f8209"
server_key_iv: "aabc11f6f47bd65baf998a23fb9cc189"
server_key_version: "encryptionkey@0"
I20260927 10:47:46.979959 20095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "3c2864e60e6e44ecbc10d219ab31008e"
format_stamp: "Formatted at 2026-09-27 10:47:46 on dist-test-slave-dzhw"
server_key: "6582a7c56c43120a1cd599bc731f8209"
server_key_iv: "aabc11f6f47bd65baf998a23fb9cc189"
server_key_version: "encryptionkey@0"
I20260927 10:47:46.983572 20095 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:47:46.985978 20109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:46.987356 20095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:47:46.987521 20095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "3c2864e60e6e44ecbc10d219ab31008e"
format_stamp: "Formatted at 2026-09-27 10:47:46 on dist-test-slave-dzhw"
server_key: "6582a7c56c43120a1cd599bc731f8209"
server_key_iv: "aabc11f6f47bd65baf998a23fb9cc189"
server_key_version: "encryptionkey@0"
I20260927 10:47:46.987641 20095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:47.026361 20095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:47.027159 20095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:47.027359 20095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:47.034600 20095 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32869
I20260927 10:47:47.034621 20161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32869 every 8 connection(s)
I20260927 10:47:47.035617 20095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:47:47.036430 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 20095
I20260927 10:47:47.036576 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:47:47.036921 392 external_mini_cluster.cc:1428] Setting key 4fa88def4669382036ffb3965935a823
I20260927 10:47:47.039351 20162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:47.045686 20162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c2864e60e6e44ecbc10d219ab31008e: Bootstrap starting.
I20260927 10:47:47.046304 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.038352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39126 (local address 127.0.98.62:32869)
0927 10:47:47.039015 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:47.039030 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:47.039065 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:47:47.039772 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:47.039789 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:47.040188 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:47.040471 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:47.040809 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.041769 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.042957 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.043420 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.043537 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.044512 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:47.044517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:47.044530 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:47:47.044916 (+ 386us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:47.044971 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:47:47.044990 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:47.045071 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:47.045229 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 10:47:47.045551 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":143,"threads_started":1}
I20260927 10:47:47.048122 20162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c2864e60e6e44ecbc10d219ab31008e: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:47.048902 20162 log.cc:824] T 00000000000000000000000000000000 P 3c2864e60e6e44ecbc10d219ab31008e: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:47.050777 20162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c2864e60e6e44ecbc10d219ab31008e: No bootstrap required, opened a new log
I20260927 10:47:47.054118 20162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c2864e60e6e44ecbc10d219ab31008e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2864e60e6e44ecbc10d219ab31008e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32869 } }
I20260927 10:47:47.054379 20162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c2864e60e6e44ecbc10d219ab31008e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:47.054478 20162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c2864e60e6e44ecbc10d219ab31008e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c2864e60e6e44ecbc10d219ab31008e, State: Initialized, Role: FOLLOWER
I20260927 10:47:47.054935 20162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c2864e60e6e44ecbc10d219ab31008e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2864e60e6e44ecbc10d219ab31008e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32869 } }
I20260927 10:47:47.055086 20162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c2864e60e6e44ecbc10d219ab31008e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:47:47.055167 20162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c2864e60e6e44ecbc10d219ab31008e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:47:47.055291 20162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c2864e60e6e44ecbc10d219ab31008e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:47.056223 20162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c2864e60e6e44ecbc10d219ab31008e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2864e60e6e44ecbc10d219ab31008e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32869 } }
I20260927 10:47:47.056559 20162 leader_election.cc:302] T 00000000000000000000000000000000 P 3c2864e60e6e44ecbc10d219ab31008e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c2864e60e6e44ecbc10d219ab31008e; no voters:
I20260927 10:47:47.056871 20162 leader_election.cc:288] T 00000000000000000000000000000000 P 3c2864e60e6e44ecbc10d219ab31008e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:47:47.057025 20167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c2864e60e6e44ecbc10d219ab31008e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:47.057269 20167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c2864e60e6e44ecbc10d219ab31008e [term 1 LEADER]: Becoming Leader. State: Replica: 3c2864e60e6e44ecbc10d219ab31008e, State: Running, Role: LEADER
I20260927 10:47:47.057588 20167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c2864e60e6e44ecbc10d219ab31008e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2864e60e6e44ecbc10d219ab31008e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32869 } }
I20260927 10:47:47.058145 20162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c2864e60e6e44ecbc10d219ab31008e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:47:47.058782 20169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c2864e60e6e44ecbc10d219ab31008e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c2864e60e6e44ecbc10d219ab31008e. Latest consensus state: current_term: 1 leader_uuid: "3c2864e60e6e44ecbc10d219ab31008e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2864e60e6e44ecbc10d219ab31008e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32869 } } }
I20260927 10:47:47.058929 20169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c2864e60e6e44ecbc10d219ab31008e [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:47.059175 20168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c2864e60e6e44ecbc10d219ab31008e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c2864e60e6e44ecbc10d219ab31008e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2864e60e6e44ecbc10d219ab31008e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32869 } } }
I20260927 10:47:47.059268 20168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c2864e60e6e44ecbc10d219ab31008e [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:47.059362 20173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:47:47.062551 20173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:47:47.068753 20173 catalog_manager.cc:1347] Generated new cluster ID: 5a2db2d49d284e51ae115783a27f7963
I20260927 10:47:47.068874 20173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:47:47.076922 20173 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:47:47.077816 20173 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:47:47.082129 20173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c2864e60e6e44ecbc10d219ab31008e: Generated new TSK 0
I20260927 10:47:47.082864 20173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:47:47.093048 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32869
--builtin_ntp_servers=127.0.98.20:35541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:47:47.205539 20186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:47.205833 20186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:47.205874 20186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:47.210338 20186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:47.210444 20186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:47.210546 20186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 10:47:47.215852 20186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:47.217054 20186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:47.218550 20186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:47.228251 20192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:47.228212 20191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:47.228209 20194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:47.228674 20186 server_base.cc:1034] running on GCE node
I20260927 10:47:47.229103 20186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:47.229777 20186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:47.230940 20186 hybrid_clock.cc:648] HybridClock initialized: now 1790506067230926 us; error 25 us; skew 500 ppm
I20260927 10:47:47.232721 20186 webserver.cc:460] Webserver started at http://127.0.98.1:34961/ using document root <none> and password file <none>
I20260927 10:47:47.233299 20186 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:47.233359 20186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:47.233577 20186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:47.235653 20186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-0/data/instance:
uuid: "f3b63d3c5e1743a5aa6bfcfcd408e364"
format_stamp: "Formatted at 2026-09-27 10:47:47 on dist-test-slave-dzhw"
server_key: "f9f0339e68973902eb5ffb3423613618"
server_key_iv: "47a653b5546eba06741cbe24e9d4d055"
server_key_version: "encryptionkey@0"
I20260927 10:47:47.236208 20186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f3b63d3c5e1743a5aa6bfcfcd408e364"
format_stamp: "Formatted at 2026-09-27 10:47:47 on dist-test-slave-dzhw"
server_key: "f9f0339e68973902eb5ffb3423613618"
server_key_iv: "47a653b5546eba06741cbe24e9d4d055"
server_key_version: "encryptionkey@0"
I20260927 10:47:47.240150 20186 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:47:47.242679 20200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:47.243958 20186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:47:47.244112 20186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-0/wal
uuid: "f3b63d3c5e1743a5aa6bfcfcd408e364"
format_stamp: "Formatted at 2026-09-27 10:47:47 on dist-test-slave-dzhw"
server_key: "f9f0339e68973902eb5ffb3423613618"
server_key_iv: "47a653b5546eba06741cbe24e9d4d055"
server_key_version: "encryptionkey@0"
I20260927 10:47:47.244234 20186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:47.259419 20186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:47.260169 20186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:47.260385 20186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:47.260982 20186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:47.261934 20186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:47.262001 20186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:47.262064 20186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:47.262109 20186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:47.271027 20186 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46271
I20260927 10:47:47.271049 20313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46271 every 8 connection(s)
I20260927 10:47:47.271966 20186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-0/data/info.pb
I20260927 10:47:47.279560 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 20186
I20260927 10:47:47.279711 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-0/wal/instance
I20260927 10:47:47.280004 392 external_mini_cluster.cc:1428] Setting key d3da19b442bd1328c175d11e094b1c32
I20260927 10:47:47.281430 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.273824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39041 (local address 127.0.98.62:32869)
0927 10:47:47.273983 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:47.273987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:47.275103 (+ 1116us) server_negotiation.cc:407] Connection header received
0927 10:47:47.276476 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:47.276478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:47.276525 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:47.276603 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:47.277476 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.278028 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.279268 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.279501 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.279624 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.280858 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:47.280866 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:47.280868 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:47:47.280879 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:47.280904 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:47:47.280912 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:47.280985 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:47.281188 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 10:47:47.281274 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:47:47.281985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32869
--builtin_ntp_servers=127.0.98.20:35541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:47:47.282550 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.274250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32869 (local address 127.0.98.1:39041)
0927 10:47:47.274950 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:47.274992 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:47.276271 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:47.276760 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:47.276767 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:47.276846 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:47.277324 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.277336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.278144 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.278146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.279137 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.279147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.279669 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.279674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.279879 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.280541 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:47:47.280543 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:47.280594 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:47:47.280691 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:47.280987 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:47.280999 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:47.281008 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 10:47:47.281157 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 10:47:47.281480 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":127,"threads_started":1}
I20260927 10:47:47.283676 20314 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32869
I20260927 10:47:47.283941 20314 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:47.284595 20314 heartbeater.cc:511] Master 127.0.98.62:32869 requested a full tablet report, sending...
I20260927 10:47:47.286007 20126 ts_manager.cc:194] Registered new tserver with Master: f3b63d3c5e1743a5aa6bfcfcd408e364 (127.0.98.1:46271)
I20260927 10:47:47.287232 20126 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:39041
W20260927 10:47:47.398265 20317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:47.398584 20317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:47.398653 20317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:47.402704 20317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:47.402763 20317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:47.402820 20317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 10:47:47.407608 20317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:47.408772 20317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:47.410131 20317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:47:47.420043 20317 server_base.cc:1034] running on GCE node
W20260927 10:47:47.420027 20323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:47.420217 20325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:47.420032 20322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:47.420648 20317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:47.421271 20317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:47.422468 20317 hybrid_clock.cc:648] HybridClock initialized: now 1790506067422432 us; error 48 us; skew 500 ppm
I20260927 10:47:47.424362 20317 webserver.cc:460] Webserver started at http://127.0.98.2:33921/ using document root <none> and password file <none>
I20260927 10:47:47.424940 20317 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:47.425005 20317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:47.425243 20317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:47.427336 20317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-1/data/instance:
uuid: "5624783d309c451bb7521d5ddb86ca88"
format_stamp: "Formatted at 2026-09-27 10:47:47 on dist-test-slave-dzhw"
server_key: "e98d9e4148d376672e108879242d347b"
server_key_iv: "ed2940b4399d70c2a13acd9d4ba6aa7e"
server_key_version: "encryptionkey@0"
I20260927 10:47:47.427887 20317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5624783d309c451bb7521d5ddb86ca88"
format_stamp: "Formatted at 2026-09-27 10:47:47 on dist-test-slave-dzhw"
server_key: "e98d9e4148d376672e108879242d347b"
server_key_iv: "ed2940b4399d70c2a13acd9d4ba6aa7e"
server_key_version: "encryptionkey@0"
I20260927 10:47:47.431597 20317 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:47:47.433993 20331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:47.435226 20317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:47:47.435345 20317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-1/wal
uuid: "5624783d309c451bb7521d5ddb86ca88"
format_stamp: "Formatted at 2026-09-27 10:47:47 on dist-test-slave-dzhw"
server_key: "e98d9e4148d376672e108879242d347b"
server_key_iv: "ed2940b4399d70c2a13acd9d4ba6aa7e"
server_key_version: "encryptionkey@0"
I20260927 10:47:47.435477 20317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:47.446785 20317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:47.447510 20317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:47.447690 20317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:47.448242 20317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:47.449206 20317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:47.449254 20317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:47.449293 20317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:47.449309 20317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:47.458573 20317 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46179
I20260927 10:47:47.458607 20444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46179 every 8 connection(s)
I20260927 10:47:47.459617 20317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-1/data/info.pb
I20260927 10:47:47.468695 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.461474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40049 (local address 127.0.98.62:32869)
0927 10:47:47.461655 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:47.461659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:47.462763 (+ 1104us) server_negotiation.cc:407] Connection header received
0927 10:47:47.464121 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:47.464124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:47.464173 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:47.464255 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:47.465181 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.465730 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.466860 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.467068 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.467147 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.468244 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:47.468247 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:47.468249 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:47:47.468261 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:47.468279 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:47:47.468287 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:47.468374 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:47.468521 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 10:47:47.468578 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 10:47:47.469429 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 20317
I20260927 10:47:47.469552 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-1/wal/instance
I20260927 10:47:47.469664 20447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.461918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32869 (local address 127.0.98.2:40049)
0927 10:47:47.462615 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:47.462655 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:47.463899 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:47.464461 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:47.464467 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:47.464547 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:47.465029 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.465041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.465846 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.465848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.466747 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.466756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.467190 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.467192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.467323 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.467983 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:47:47.467984 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:47.468025 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:47:47.468088 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:47.468383 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:47.468389 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:47.468393 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 10:47:47.468499 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 10:47:47.468716 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":109,"threads_started":1}
I20260927 10:47:47.469883 392 external_mini_cluster.cc:1428] Setting key c3a7b46b62f95c4d043aa2530e071e51
I20260927 10:47:47.470861 20445 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32869
I20260927 10:47:47.471113 20445 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:47.471741 20445 heartbeater.cc:511] Master 127.0.98.62:32869 requested a full tablet report, sending...
I20260927 10:47:47.472090 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32869
--builtin_ntp_servers=127.0.98.20:35541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:47:47.472950 20126 ts_manager.cc:194] Registered new tserver with Master: 5624783d309c451bb7521d5ddb86ca88 (127.0.98.2:46179)
I20260927 10:47:47.473811 20126 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40049
W20260927 10:47:47.582407 20448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:47.582726 20448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:47.582765 20448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:47.586870 20448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:47.586935 20448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:47.586992 20448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 10:47:47.591866 20448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:47.592895 20448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:47.594187 20448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:47:47.603585 20448 server_base.cc:1034] running on GCE node
W20260927 10:47:47.603638 20453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:47.603735 20454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:47.603650 20456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:47.604195 20448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:47.604830 20448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:47.605991 20448 hybrid_clock.cc:648] HybridClock initialized: now 1790506067605977 us; error 28 us; skew 500 ppm
I20260927 10:47:47.612823 20448 webserver.cc:460] Webserver started at http://127.0.98.3:36811/ using document root <none> and password file <none>
I20260927 10:47:47.613600 20448 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:47.613678 20448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:47.613969 20448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:47.616325 20448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-2/data/instance:
uuid: "e35b1090804e452da3da912d4f827379"
format_stamp: "Formatted at 2026-09-27 10:47:47 on dist-test-slave-dzhw"
server_key: "9e7e886ae9ffcbfdb4659ba1e051607c"
server_key_iv: "01fe6e3e0c800b5005919eacdbf689a5"
server_key_version: "encryptionkey@0"
I20260927 10:47:47.616914 20448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e35b1090804e452da3da912d4f827379"
format_stamp: "Formatted at 2026-09-27 10:47:47 on dist-test-slave-dzhw"
server_key: "9e7e886ae9ffcbfdb4659ba1e051607c"
server_key_iv: "01fe6e3e0c800b5005919eacdbf689a5"
server_key_version: "encryptionkey@0"
I20260927 10:47:47.620872 20448 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 10:47:47.623484 20462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:47.624811 20448 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:47:47.624951 20448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-2/wal
uuid: "e35b1090804e452da3da912d4f827379"
format_stamp: "Formatted at 2026-09-27 10:47:47 on dist-test-slave-dzhw"
server_key: "9e7e886ae9ffcbfdb4659ba1e051607c"
server_key_iv: "01fe6e3e0c800b5005919eacdbf689a5"
server_key_version: "encryptionkey@0"
I20260927 10:47:47.625051 20448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:47.656327 20448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:47.656996 20448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:47.657151 20448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:47.657737 20448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:47.658825 20448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:47.658895 20448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:47.658972 20448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:47.659003 20448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:47.667896 20448 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36517
I20260927 10:47:47.667925 20575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36517 every 8 connection(s)
I20260927 10:47:47.668820 20448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-2/data/info.pb
I20260927 10:47:47.678076 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.670740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44981 (local address 127.0.98.62:32869)
0927 10:47:47.670918 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:47.670923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:47.671961 (+ 1038us) server_negotiation.cc:407] Connection header received
0927 10:47:47.673338 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:47.673341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:47.673393 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:47.673482 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:47.674382 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.675076 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.676191 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.676456 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.676553 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.677652 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:47.677655 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:47.677657 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:47:47.677670 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:47.677690 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:47:47.677697 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:47.677766 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:47.677897 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 10:47:47.677950 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:47:47.679014 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.671168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32869 (local address 127.0.98.3:44981)
0927 10:47:47.671793 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:47.671837 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:47.673078 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:47.673644 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:47.673651 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:47.673731 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:47.674217 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.674228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.675192 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.675194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.676067 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.676076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.676595 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.676598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.676736 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.677390 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:47:47.677392 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:47.677432 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:47:47.677496 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:47.677775 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:47.677781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:47.677786 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 10:47:47.677880 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 10:47:47.678098 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":112,"threads_started":1}
I20260927 10:47:47.679044 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 20448
I20260927 10:47:47.679144 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-2/wal/instance
I20260927 10:47:47.679417 392 external_mini_cluster.cc:1428] Setting key b454a240c3d5e1d79e4fb18bca7b4a56
I20260927 10:47:47.680187 20576 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32869
I20260927 10:47:47.680447 20576 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:47.681023 20576 heartbeater.cc:511] Master 127.0.98.62:32869 requested a full tablet report, sending...
I20260927 10:47:47.681994 20126 ts_manager.cc:194] Registered new tserver with Master: e35b1090804e452da3da912d4f827379 (127.0.98.3:36517)
I20260927 10:47:47.682626 20126 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:44981
I20260927 10:47:47.693702 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:47:47.699313 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.695786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39130 (local address 127.0.98.62:32869)
0927 10:47:47.695916 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:47.695919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:47.696058 (+ 139us) server_negotiation.cc:407] Connection header received
0927 10:47:47.696166 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:47.696169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:47.696207 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:47.696275 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:47.696586 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.697089 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.697904 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.698111 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.698186 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.698999 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:47.699002 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:47.699004 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:47:47.699015 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:47.699032 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:47:47.699039 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:47.699096 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:47.699162 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 10:47:47.699206 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:47:47.701970 20126 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:47:47.704990 20126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:47:47.719534 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.713372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48012 (local address 127.0.98.2:46179)
0927 10:47:47.713807 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:47.713819 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:47.714079 (+ 260us) server_negotiation.cc:407] Connection header received
0927 10:47:47.714163 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:47.714171 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:47.714381 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:47.714571 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:47.715049 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.716148 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.717695 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.718515 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.718644 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.718841 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:47.719252 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 10:47:47.719355 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":109,"threads_started":1}
I20260927 10:47:47.719563 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.712921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36517 (local address 127.0.0.1:52324)
0927 10:47:47.713652 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:47.713703 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:47.713833 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:47.714624 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:47.714630 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:47.714651 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:47.714785 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.714792 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.716303 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.716306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.717732 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.717743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.718705 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.718708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.718940 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.719077 (+ 137us) client_negotiation.cc:769] Sending connection context
0927 10:47:47.719219 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 10:47:47.719360 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":208,"thread_start_us":76,"threads_started":1}
I20260927 10:47:47.720026 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.713079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52324 (local address 127.0.98.3:36517)
0927 10:47:47.714141 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:47.714152 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:47.714176 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:47:47.714252 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:47.714256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:47.714462 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:47.714635 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:47.714921 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.716148 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.717853 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.718587 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.718681 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.718841 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:47.719792 (+ 951us) server_negotiation.cc:299] Negotiation successful
0927 10:47:47.719889 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":954,"thread_start_us":162,"threads_started":1}
I20260927 10:47:47.720438 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.713637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46271 (local address 127.0.0.1:38032)
0927 10:47:47.714063 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:47.714076 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:47.714152 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:47.715286 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:47.715289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:47.715306 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:47.715382 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.715388 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.716388 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.716391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.717732 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.717743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.720022 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.720025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.720094 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.720142 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 10:47:47.720230 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 10:47:47.720304 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":153,"thread_start_us":88,"threads_started":1}
I20260927 10:47:47.719563 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.713018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46179 (local address 127.0.0.1:48012)
0927 10:47:47.713851 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:47.713866 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:47.713980 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:47.714722 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:47.714728 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:47.714743 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:47.714921 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.714931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.716269 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.716273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.717559 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.717568 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.718686 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.718691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.718960 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.719077 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 10:47:47.719212 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 10:47:47.719356 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":162,"thread_start_us":92,"threads_started":1}
I20260927 10:47:47.721071 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.714063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38032 (local address 127.0.98.1:46271)
0927 10:47:47.714912 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:47.714924 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:47.714946 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:47:47.715021 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:47.715029 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:47.715171 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:47.715306 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:47.715484 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.716283 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.718899 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.719543 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.719656 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.719836 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:47.720791 (+ 955us) server_negotiation.cc:299] Negotiation successful
0927 10:47:47.720899 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":115,"threads_started":1}
I20260927 10:47:47.723241 20379 tablet_service.cc:1467] Processing CreateTablet for tablet da68f8b9f43a4bba873d60bfea08b530 (DEFAULT_TABLE table=test-table [id=7cced3efa2184a43869a324133c18645]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:47.723241 20510 tablet_service.cc:1467] Processing CreateTablet for tablet da68f8b9f43a4bba873d60bfea08b530 (DEFAULT_TABLE table=test-table [id=7cced3efa2184a43869a324133c18645]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:47.723582 20248 tablet_service.cc:1467] Processing CreateTablet for tablet da68f8b9f43a4bba873d60bfea08b530 (DEFAULT_TABLE table=test-table [id=7cced3efa2184a43869a324133c18645]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:47.724227 20379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da68f8b9f43a4bba873d60bfea08b530. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:47.724236 20510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da68f8b9f43a4bba873d60bfea08b530. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:47.724417 20248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da68f8b9f43a4bba873d60bfea08b530. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:47.731980 20595 tablet_bootstrap.cc:492] T da68f8b9f43a4bba873d60bfea08b530 P e35b1090804e452da3da912d4f827379: Bootstrap starting.
I20260927 10:47:47.731412 20594 tablet_bootstrap.cc:492] T da68f8b9f43a4bba873d60bfea08b530 P 5624783d309c451bb7521d5ddb86ca88: Bootstrap starting.
I20260927 10:47:47.733873 20595 tablet_bootstrap.cc:654] T da68f8b9f43a4bba873d60bfea08b530 P e35b1090804e452da3da912d4f827379: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:47.733878 20594 tablet_bootstrap.cc:654] T da68f8b9f43a4bba873d60bfea08b530 P 5624783d309c451bb7521d5ddb86ca88: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:47.734678 20594 log.cc:824] T da68f8b9f43a4bba873d60bfea08b530 P 5624783d309c451bb7521d5ddb86ca88: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:47.734685 20595 log.cc:824] T da68f8b9f43a4bba873d60bfea08b530 P e35b1090804e452da3da912d4f827379: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:47.734768 20593 tablet_bootstrap.cc:492] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364: Bootstrap starting.
I20260927 10:47:47.736915 20595 tablet_bootstrap.cc:492] T da68f8b9f43a4bba873d60bfea08b530 P e35b1090804e452da3da912d4f827379: No bootstrap required, opened a new log
I20260927 10:47:47.737010 20594 tablet_bootstrap.cc:492] T da68f8b9f43a4bba873d60bfea08b530 P 5624783d309c451bb7521d5ddb86ca88: No bootstrap required, opened a new log
I20260927 10:47:47.737057 20593 tablet_bootstrap.cc:654] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:47.737149 20595 ts_tablet_manager.cc:1397] T da68f8b9f43a4bba873d60bfea08b530 P e35b1090804e452da3da912d4f827379: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:47:47.737198 20594 ts_tablet_manager.cc:1397] T da68f8b9f43a4bba873d60bfea08b530 P 5624783d309c451bb7521d5ddb86ca88: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260927 10:47:47.737941 20593 log.cc:824] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:47.740201 20593 tablet_bootstrap.cc:492] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364: No bootstrap required, opened a new log
I20260927 10:47:47.740408 20593 ts_tablet_manager.cc:1397] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:47:47.741459 20595 raft_consensus.cc:348] T da68f8b9f43a4bba873d60bfea08b530 P e35b1090804e452da3da912d4f827379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35b1090804e452da3da912d4f827379" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36517 } } peers { permanent_uuid: "5624783d309c451bb7521d5ddb86ca88" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46179 } } peers { permanent_uuid: "f3b63d3c5e1743a5aa6bfcfcd408e364" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46271 } }
I20260927 10:47:47.741566 20594 raft_consensus.cc:348] T da68f8b9f43a4bba873d60bfea08b530 P 5624783d309c451bb7521d5ddb86ca88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35b1090804e452da3da912d4f827379" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36517 } } peers { permanent_uuid: "5624783d309c451bb7521d5ddb86ca88" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46179 } } peers { permanent_uuid: "f3b63d3c5e1743a5aa6bfcfcd408e364" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46271 } }
I20260927 10:47:47.741761 20595 raft_consensus.cc:374] T da68f8b9f43a4bba873d60bfea08b530 P e35b1090804e452da3da912d4f827379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:47.741806 20594 raft_consensus.cc:374] T da68f8b9f43a4bba873d60bfea08b530 P 5624783d309c451bb7521d5ddb86ca88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:47.741834 20595 raft_consensus.cc:729] T da68f8b9f43a4bba873d60bfea08b530 P e35b1090804e452da3da912d4f827379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35b1090804e452da3da912d4f827379, State: Initialized, Role: FOLLOWER
I20260927 10:47:47.741865 20594 raft_consensus.cc:729] T da68f8b9f43a4bba873d60bfea08b530 P 5624783d309c451bb7521d5ddb86ca88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5624783d309c451bb7521d5ddb86ca88, State: Initialized, Role: FOLLOWER
I20260927 10:47:47.742329 20595 consensus_queue.cc:260] T da68f8b9f43a4bba873d60bfea08b530 P e35b1090804e452da3da912d4f827379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35b1090804e452da3da912d4f827379" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36517 } } peers { permanent_uuid: "5624783d309c451bb7521d5ddb86ca88" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46179 } } peers { permanent_uuid: "f3b63d3c5e1743a5aa6bfcfcd408e364" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46271 } }
I20260927 10:47:47.742367 20594 consensus_queue.cc:260] T da68f8b9f43a4bba873d60bfea08b530 P 5624783d309c451bb7521d5ddb86ca88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35b1090804e452da3da912d4f827379" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36517 } } peers { permanent_uuid: "5624783d309c451bb7521d5ddb86ca88" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46179 } } peers { permanent_uuid: "f3b63d3c5e1743a5aa6bfcfcd408e364" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46271 } }
I20260927 10:47:47.743068 20576 heartbeater.cc:503] Master 127.0.98.62:32869 was elected leader, sending a full tablet report...
I20260927 10:47:47.743444 20594 ts_tablet_manager.cc:1428] T da68f8b9f43a4bba873d60bfea08b530 P 5624783d309c451bb7521d5ddb86ca88: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:47:47.743808 20445 heartbeater.cc:503] Master 127.0.98.62:32869 was elected leader, sending a full tablet report...
I20260927 10:47:47.744519 20595 ts_tablet_manager.cc:1428] T da68f8b9f43a4bba873d60bfea08b530 P e35b1090804e452da3da912d4f827379: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 10:47:47.744643 20593 raft_consensus.cc:348] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35b1090804e452da3da912d4f827379" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36517 } } peers { permanent_uuid: "5624783d309c451bb7521d5ddb86ca88" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46179 } } peers { permanent_uuid: "f3b63d3c5e1743a5aa6bfcfcd408e364" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46271 } }
I20260927 10:47:47.744884 20593 raft_consensus.cc:374] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:47.744946 20593 raft_consensus.cc:729] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b63d3c5e1743a5aa6bfcfcd408e364, State: Initialized, Role: FOLLOWER
I20260927 10:47:47.745395 20593 consensus_queue.cc:260] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35b1090804e452da3da912d4f827379" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36517 } } peers { permanent_uuid: "5624783d309c451bb7521d5ddb86ca88" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46179 } } peers { permanent_uuid: "f3b63d3c5e1743a5aa6bfcfcd408e364" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46271 } }
I20260927 10:47:47.746035 20314 heartbeater.cc:503] Master 127.0.98.62:32869 was elected leader, sending a full tablet report...
I20260927 10:47:47.746312 20593 ts_tablet_manager.cc:1428] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:47:47.752919 20601 raft_consensus.cc:482] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:47:47.753115 20601 raft_consensus.cc:504] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35b1090804e452da3da912d4f827379" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36517 } } peers { permanent_uuid: "5624783d309c451bb7521d5ddb86ca88" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46179 } } peers { permanent_uuid: "f3b63d3c5e1743a5aa6bfcfcd408e364" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46271 } }
I20260927 10:47:47.754215 20601 leader_election.cc:288] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e35b1090804e452da3da912d4f827379 (127.0.98.3:36517), 5624783d309c451bb7521d5ddb86ca88 (127.0.98.2:46179)
I20260927 10:47:47.758201 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.754566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45213 (local address 127.0.98.2:46179)
0927 10:47:47.754678 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:47.754683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:47.754700 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:47:47.754804 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:47.754807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:47.754846 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:47.754918 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:47.755212 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.755910 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.757143 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.757654 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.757768 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.757830 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:47.758035 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 10:47:47.758103 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:47:47.758201 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.754493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46179 (local address 127.0.98.1:45213)
0927 10:47:47.754605 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:47.754622 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:47.754707 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:47.754955 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:47.754958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:47.754975 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:47.755102 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.755108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.756018 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.756021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.757034 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.757043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.757769 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.757772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.757879 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.757901 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:47:47.757991 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 10:47:47.758065 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":28}
I20260927 10:47:47.758376 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.754566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38137 (local address 127.0.98.3:36517)
0927 10:47:47.754678 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:47.754683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:47.755225 (+ 542us) server_negotiation.cc:407] Connection header received
0927 10:47:47.755341 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:47.755344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:47.755385 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:47.755442 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:47.755708 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.756175 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.757415 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.757797 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.757919 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.757994 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:47.758195 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 10:47:47.758257 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:47:47.758397 20602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.754496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36517 (local address 127.0.98.1:38137)
0927 10:47:47.755117 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:47.755129 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:47.755243 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:47.755475 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:47.755478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:47.755495 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:47.755606 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.755611 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.756281 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.756286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.757293 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.757303 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.757931 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.757934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.758071 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.758095 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 10:47:47.758180 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 10:47:47.758245 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":290,"thread_start_us":136,"threads_started":1}
I20260927 10:47:47.759079 20530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da68f8b9f43a4bba873d60bfea08b530" candidate_uuid: "f3b63d3c5e1743a5aa6bfcfcd408e364" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35b1090804e452da3da912d4f827379" is_pre_election: true
I20260927 10:47:47.759156 20399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da68f8b9f43a4bba873d60bfea08b530" candidate_uuid: "f3b63d3c5e1743a5aa6bfcfcd408e364" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5624783d309c451bb7521d5ddb86ca88" is_pre_election: true
I20260927 10:47:47.759411 20530 raft_consensus.cc:2454] T da68f8b9f43a4bba873d60bfea08b530 P e35b1090804e452da3da912d4f827379 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3b63d3c5e1743a5aa6bfcfcd408e364 in term 0.
I20260927 10:47:47.759411 20399 raft_consensus.cc:2454] T da68f8b9f43a4bba873d60bfea08b530 P 5624783d309c451bb7521d5ddb86ca88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3b63d3c5e1743a5aa6bfcfcd408e364 in term 0.
I20260927 10:47:47.759895 20202 leader_election.cc:302] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5624783d309c451bb7521d5ddb86ca88, e35b1090804e452da3da912d4f827379, f3b63d3c5e1743a5aa6bfcfcd408e364; no voters:
I20260927 10:47:47.760192 20601 raft_consensus.cc:2781] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:47:47.760275 20601 raft_consensus.cc:482] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:47:47.760334 20601 raft_consensus.cc:3037] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:47.761358 20601 raft_consensus.cc:504] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35b1090804e452da3da912d4f827379" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36517 } } peers { permanent_uuid: "5624783d309c451bb7521d5ddb86ca88" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46179 } } peers { permanent_uuid: "f3b63d3c5e1743a5aa6bfcfcd408e364" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46271 } }
I20260927 10:47:47.761695 20601 leader_election.cc:288] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364 [CANDIDATE]: Term 1 election: Requested vote from peers e35b1090804e452da3da912d4f827379 (127.0.98.3:36517), 5624783d309c451bb7521d5ddb86ca88 (127.0.98.2:46179)
I20260927 10:47:47.762053 20530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da68f8b9f43a4bba873d60bfea08b530" candidate_uuid: "f3b63d3c5e1743a5aa6bfcfcd408e364" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35b1090804e452da3da912d4f827379"
I20260927 10:47:47.762073 20399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da68f8b9f43a4bba873d60bfea08b530" candidate_uuid: "f3b63d3c5e1743a5aa6bfcfcd408e364" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5624783d309c451bb7521d5ddb86ca88"
I20260927 10:47:47.762166 20530 raft_consensus.cc:3037] T da68f8b9f43a4bba873d60bfea08b530 P e35b1090804e452da3da912d4f827379 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:47.762216 20399 raft_consensus.cc:3037] T da68f8b9f43a4bba873d60bfea08b530 P 5624783d309c451bb7521d5ddb86ca88 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:47.763270 20530 raft_consensus.cc:2454] T da68f8b9f43a4bba873d60bfea08b530 P e35b1090804e452da3da912d4f827379 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3b63d3c5e1743a5aa6bfcfcd408e364 in term 1.
I20260927 10:47:47.763381 20399 raft_consensus.cc:2454] T da68f8b9f43a4bba873d60bfea08b530 P 5624783d309c451bb7521d5ddb86ca88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3b63d3c5e1743a5aa6bfcfcd408e364 in term 1.
I20260927 10:47:47.763620 20201 leader_election.cc:302] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e35b1090804e452da3da912d4f827379, f3b63d3c5e1743a5aa6bfcfcd408e364; no voters:
I20260927 10:47:47.763827 20601 raft_consensus.cc:2781] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:47.764025 20601 raft_consensus.cc:686] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364 [term 1 LEADER]: Becoming Leader. State: Replica: f3b63d3c5e1743a5aa6bfcfcd408e364, State: Running, Role: LEADER
I20260927 10:47:47.764351 20601 consensus_queue.cc:237] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35b1090804e452da3da912d4f827379" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36517 } } peers { permanent_uuid: "5624783d309c451bb7521d5ddb86ca88" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46179 } } peers { permanent_uuid: "f3b63d3c5e1743a5aa6bfcfcd408e364" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46271 } }
I20260927 10:47:47.767637 20124 catalog_manager.cc:5495] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3b63d3c5e1743a5aa6bfcfcd408e364 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f3b63d3c5e1743a5aa6bfcfcd408e364" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35b1090804e452da3da912d4f827379" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5624783d309c451bb7521d5ddb86ca88" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3b63d3c5e1743a5aa6bfcfcd408e364" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46271 } health_report { overall_health: HEALTHY } } }
W20260927 10:47:47.773104 20315 tablet.cc:2369] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:47:47.788398 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.785035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38048 (local address 127.0.98.1:46271)
0927 10:47:47.785167 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:47.785171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:47.785184 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:47:47.785287 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:47.785289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:47.785343 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:47.785439 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:47:47.785639 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.786132 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.787000 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.787210 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.787290 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.787541 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:47:47.787952 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:47:47.788040 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:47.788154 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 10:47:47.788200 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:47:47.794454 20399 raft_consensus.cc:1261] T da68f8b9f43a4bba873d60bfea08b530 P 5624783d309c451bb7521d5ddb86ca88 [term 1 FOLLOWER]: Refusing update from remote peer f3b63d3c5e1743a5aa6bfcfcd408e364: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:47:47.794497 20530 raft_consensus.cc:1261] T da68f8b9f43a4bba873d60bfea08b530 P e35b1090804e452da3da912d4f827379 [term 1 FOLLOWER]: Refusing update from remote peer f3b63d3c5e1743a5aa6bfcfcd408e364: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:47:47.795198 20601 consensus_queue.cc:1035] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e35b1090804e452da3da912d4f827379" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:47:47.795465 20603 consensus_queue.cc:1035] T da68f8b9f43a4bba873d60bfea08b530 P f3b63d3c5e1743a5aa6bfcfcd408e364 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5624783d309c451bb7521d5ddb86ca88" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:47:47.803407 20606 mvcc.cc:205] Tried to move back new op lower bound from 7333912853678751744 to 7333912853565566976. Current Snapshot: MvccSnapshot[applied={T|T < 7333912853678751744}]
I20260927 10:47:47.804191 20607 mvcc.cc:205] Tried to move back new op lower bound from 7333912853678751744 to 7333912853565566976. Current Snapshot: MvccSnapshot[applied={T|T < 7333912853678751744}]
I20260927 10:47:47.813059 20125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:47:47.813266 20125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:47:47.816047 20125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c2864e60e6e44ecbc10d219ab31008e: Sending DeleteTablet for 3 replicas of tablet da68f8b9f43a4bba873d60bfea08b530
I20260927 10:47:47.816876 20510 tablet_service.cc:1514] Processing DeleteTablet for tablet da68f8b9f43a4bba873d60bfea08b530 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:47 UTC) from {username='slave'} at 127.0.0.1:52324
I20260927 10:47:47.816929 20248 tablet_service.cc:1514] Processing DeleteTablet for tablet da68f8b9f43a4bba873d60bfea08b530 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:47 UTC) from {username='slave'} at 127.0.0.1:38032
I20260927 10:47:47.817019 20379 tablet_service.cc:1514] Processing DeleteTablet for tablet da68f8b9f43a4bba873d60bfea08b530 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:47 UTC) from {username='slave'} at 127.0.0.1:48012
I20260927 10:47:47.817483 20616 tablet_replica.cc:330] T da68f8b9f43a4bba873d60bfea08b530 P e35b1090804e452da3da912d4f827379: stopping tablet replica
I20260927 10:47:47.817817 20616 raft_consensus.cc:2229] T da68f8b9f43a4bba873d60bfea08b530 P e35b1090804e452da3da912d4f827379 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:47:47.818127 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 20186
I20260927 10:47:47.819123 20618 tablet_replica.cc:330] T da68f8b9f43a4bba873d60bfea08b530 P 5624783d309c451bb7521d5ddb86ca88: stopping tablet replica
I20260927 10:47:47.819412 20618 raft_consensus.cc:2229] T da68f8b9f43a4bba873d60bfea08b530 P 5624783d309c451bb7521d5ddb86ca88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:47:47.819689 20618 raft_consensus.cc:2258] T da68f8b9f43a4bba873d60bfea08b530 P 5624783d309c451bb7521d5ddb86ca88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:47.818179 20616 raft_consensus.cc:2258] T da68f8b9f43a4bba873d60bfea08b530 P e35b1090804e452da3da912d4f827379 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:47.821290 20618 ts_tablet_manager.cc:1905] T da68f8b9f43a4bba873d60bfea08b530 P 5624783d309c451bb7521d5ddb86ca88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:47:47.821633 20616 ts_tablet_manager.cc:1905] T da68f8b9f43a4bba873d60bfea08b530 P e35b1090804e452da3da912d4f827379: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:47:47.823602 20618 ts_tablet_manager.cc:1918] T da68f8b9f43a4bba873d60bfea08b530 P 5624783d309c451bb7521d5ddb86ca88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:47:47.823727 20618 log.cc:1196] T da68f8b9f43a4bba873d60bfea08b530 P 5624783d309c451bb7521d5ddb86ca88: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-1/wal/wals/da68f8b9f43a4bba873d60bfea08b530
I20260927 10:47:47.824031 20618 ts_tablet_manager.cc:1939] T da68f8b9f43a4bba873d60bfea08b530 P 5624783d309c451bb7521d5ddb86ca88: Deleting consensus metadata
I20260927 10:47:47.824318 20616 ts_tablet_manager.cc:1918] T da68f8b9f43a4bba873d60bfea08b530 P e35b1090804e452da3da912d4f827379: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:47:47.824417 20616 log.cc:1196] T da68f8b9f43a4bba873d60bfea08b530 P e35b1090804e452da3da912d4f827379: Deleting WAL directory at /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505812134190-392-0/minicluster-data/ts-2/wal/wals/da68f8b9f43a4bba873d60bfea08b530
I20260927 10:47:47.824723 20616 ts_tablet_manager.cc:1939] T da68f8b9f43a4bba873d60bfea08b530 P e35b1090804e452da3da912d4f827379: Deleting consensus metadata
I20260927 10:47:47.825345 20111 catalog_manager.cc:4841] TS 5624783d309c451bb7521d5ddb86ca88 (127.0.98.2:46179): tablet da68f8b9f43a4bba873d60bfea08b530 (table test-table [id=7cced3efa2184a43869a324133c18645]) successfully deleted
I20260927 10:47:47.825836 20110 catalog_manager.cc:4841] TS e35b1090804e452da3da912d4f827379 (127.0.98.3:36517): tablet da68f8b9f43a4bba873d60bfea08b530 (table test-table [id=7cced3efa2184a43869a324133c18645]) successfully deleted
W20260927 10:47:47.826087 20110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46271 (error 108)
I20260927 10:47:47.826257 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 20317
I20260927 10:47:47.828111 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.827772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46271 (local address 127.0.0.1:38056)
0927 10:47:47.827935 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:47.828026 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 10:47:47.828292 20110 catalog_manager.cc:4584] TS f3b63d3c5e1743a5aa6bfcfcd408e364 (127.0.98.1:46271): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet da68f8b9f43a4bba873d60bfea08b530: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46271: connect: Connection refused (error 111)
I20260927 10:47:47.832700 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 20448
I20260927 10:47:47.838733 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 20095
2026-09-27T10:47:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1027 ms)
[----------] 6 tests from EncryptionPolicyTest (10407 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:47:47 dist-test-slave-dzhw krb5kdc[20622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:47:47 dist-test-slave-dzhw krb5kdc[20622](info): set up 2 sockets
Sep 27 10:47:47 dist-test-slave-dzhw krb5kdc[20622](info): commencing operation
krb5kdc: starting...
W20260927 10:47:48.878576 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:47:48 dist-test-slave-dzhw krb5kdc[20622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:47:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:47:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:47:49.002534 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36853
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:49.111639 20635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:49.111961 20635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:49.112038 20635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:49.116076 20635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:49.116164 20635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:49.116210 20635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:49.116259 20635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:49.116295 20635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:49.121671 20635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36853
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:49.122848 20635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:49.123857 20635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:49.132143 20640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:49.132347 20635 server_base.cc:1034] running on GCE node
W20260927 10:47:49.132170 20643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:49.132148 20641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:49.133001 20635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:49.134140 20635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:49.135349 20635 hybrid_clock.cc:648] HybridClock initialized: now 1790506069135309 us; error 52 us; skew 500 ppm
Sep 27 10:47:49 dist-test-slave-dzhw krb5kdc[20622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:49.138195 20635 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:47:49.139150 20635 webserver.cc:460] Webserver started at http://127.0.0.1:35331/ using document root <none> and password file <none>
I20260927 10:47:49.139719 20635 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:49.139809 20635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:49.140031 20635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:49.142091 20635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "c6c696fce66449c3850a5a6ff579b9dd"
format_stamp: "Formatted at 2026-09-27 10:47:49 on dist-test-slave-dzhw"
server_key: "beee757b19e6755cc5c0efae01dbc804"
server_key_iv: "1d1aaa96cd37a6da240da28609ac5e5b"
server_key_version: "encryptionkey@0"
I20260927 10:47:49.142699 20635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "c6c696fce66449c3850a5a6ff579b9dd"
format_stamp: "Formatted at 2026-09-27 10:47:49 on dist-test-slave-dzhw"
server_key: "beee757b19e6755cc5c0efae01dbc804"
server_key_iv: "1d1aaa96cd37a6da240da28609ac5e5b"
server_key_version: "encryptionkey@0"
I20260927 10:47:49.146466 20635 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:47:49.148921 20650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:49.150234 20635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:47:49.150390 20635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "c6c696fce66449c3850a5a6ff579b9dd"
format_stamp: "Formatted at 2026-09-27 10:47:49 on dist-test-slave-dzhw"
server_key: "beee757b19e6755cc5c0efae01dbc804"
server_key_iv: "1d1aaa96cd37a6da240da28609ac5e5b"
server_key_version: "encryptionkey@0"
I20260927 10:47:49.150563 20635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:49.171130 20635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:49.174386 20635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:49.174609 20635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:49.182961 20635 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36853
I20260927 10:47:49.182965 20702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36853 every 8 connection(s)
I20260927 10:47:49.184293 20635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:47:49.188601 20703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:49.189150 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 20635
I20260927 10:47:49.189316 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:47:49.189685 392 external_mini_cluster.cc:1428] Setting key 94c45f5133cc5f76efeac5842bf1e22e
I20260927 10:47:49.196762 20703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6c696fce66449c3850a5a6ff579b9dd: Bootstrap starting.
I20260927 10:47:49.200026 20703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6c696fce66449c3850a5a6ff579b9dd: Neither blocks nor log segments found. Creating new log.
Sep 27 10:47:49 dist-test-slave-dzhw krb5kdc[20622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 10:47:49.200891 20703 log.cc:824] T 00000000000000000000000000000000 P c6c696fce66449c3850a5a6ff579b9dd: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:49.203284 20703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6c696fce66449c3850a5a6ff579b9dd: No bootstrap required, opened a new log
I20260927 10:47:49.206986 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:49.191411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54911 (local address 127.0.0.1:36853)
0927 10:47:49.192075 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:49.192090 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:49.192142 (+ 52us) server_negotiation.cc:407] Connection header received
0927 10:47:49.192816 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:49.192832 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:49.193199 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:49.193733 (+ 534us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:49.194673 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:49.195813 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:49.196729 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:49.197451 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:49.197665 (+ 214us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:49.201041 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:49.201047 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:49.201064 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:49.201095 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:49.203252 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:49.203952 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:49.203958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:49.203964 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:49.204084 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:49.204547 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:49.204551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:49.204553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:49.205070 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:49.205313 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:49.205693 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 10:47:49.205965 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":132,"threads_started":1}
I20260927 10:47:49.207981 20703 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6c696fce66449c3850a5a6ff579b9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c696fce66449c3850a5a6ff579b9dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36853 } }
I20260927 10:47:49.208262 20703 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6c696fce66449c3850a5a6ff579b9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:49.208319 20703 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6c696fce66449c3850a5a6ff579b9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c696fce66449c3850a5a6ff579b9dd, State: Initialized, Role: FOLLOWER
I20260927 10:47:49.208786 20703 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6c696fce66449c3850a5a6ff579b9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c696fce66449c3850a5a6ff579b9dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36853 } }
I20260927 10:47:49.208940 20703 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6c696fce66449c3850a5a6ff579b9dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:47:49.208997 20703 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6c696fce66449c3850a5a6ff579b9dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:47:49.209077 20703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6c696fce66449c3850a5a6ff579b9dd [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:49.210314 20703 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6c696fce66449c3850a5a6ff579b9dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c696fce66449c3850a5a6ff579b9dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36853 } }
I20260927 10:47:49.210691 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:47:49.210767 20703 leader_election.cc:302] T 00000000000000000000000000000000 P c6c696fce66449c3850a5a6ff579b9dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6c696fce66449c3850a5a6ff579b9dd; no voters:
I20260927 10:47:49.211202 20703 leader_election.cc:288] T 00000000000000000000000000000000 P c6c696fce66449c3850a5a6ff579b9dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:47:49.211319 20708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6c696fce66449c3850a5a6ff579b9dd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:49.211560 20708 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6c696fce66449c3850a5a6ff579b9dd [term 1 LEADER]: Becoming Leader. State: Replica: c6c696fce66449c3850a5a6ff579b9dd, State: Running, Role: LEADER
I20260927 10:47:49.211906 20708 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6c696fce66449c3850a5a6ff579b9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c696fce66449c3850a5a6ff579b9dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36853 } }
I20260927 10:47:49.212451 20703 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6c696fce66449c3850a5a6ff579b9dd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:47:49.213171 20708 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6c696fce66449c3850a5a6ff579b9dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6c696fce66449c3850a5a6ff579b9dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c696fce66449c3850a5a6ff579b9dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36853 } } }
I20260927 10:47:49.213301 20708 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6c696fce66449c3850a5a6ff579b9dd [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:49.213513 20708 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6c696fce66449c3850a5a6ff579b9dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6c696fce66449c3850a5a6ff579b9dd. Latest consensus state: current_term: 1 leader_uuid: "c6c696fce66449c3850a5a6ff579b9dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c696fce66449c3850a5a6ff579b9dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36853 } } }
I20260927 10:47:49.213603 20708 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6c696fce66449c3850a5a6ff579b9dd [sys.catalog]: This master's current role is: LEADER
W20260927 10:47:49.217401 20728 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6c696fce66449c3850a5a6ff579b9dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:47:49.217482 20728 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:47:49.217885 20722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:47:49.218752 20722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:47:49.222555 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:49.212962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33285 (local address 127.0.0.1:36853)
0927 10:47:49.213069 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:49.213073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:49.213470 (+ 397us) server_negotiation.cc:407] Connection header received
0927 10:47:49.213514 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:49.213517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:49.213567 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:49.213642 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:49.214616 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:49.215346 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:49.216415 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:49.216754 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:49.216834 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:49.219027 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:49.219031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:49.219035 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:49.219063 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:49.220749 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:49.221247 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:49.221251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:49.221253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:49.221336 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:49.221661 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:49.221664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:49.221666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:49.221845 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:49.222004 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:49.222268 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 10:47:49.222359 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":41}
I20260927 10:47:49.224713 20722 catalog_manager.cc:1347] Generated new cluster ID: 31e009309c774dc685eedd023f29ec60
I20260927 10:47:49.224769 20722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:47:49.233143 20722 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:47:49.234141 20722 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:47:49.245383 20722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6c696fce66449c3850a5a6ff579b9dd: Generated new TSK 0
I20260927 10:47:49.246134 20722 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:47:49.250746 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 20635
W20260927 10:47:49.256297 20620 connection.cc:535] client connection to 127.0.0.1:36853 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36853): Connection reset by peer (error 104)
2026-09-27T10:47:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:47:49 dist-test-slave-dzhw krb5kdc[20734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:47:49 dist-test-slave-dzhw krb5kdc[20734](info): set up 2 sockets
Sep 27 10:47:49 dist-test-slave-dzhw krb5kdc[20734](info): commencing operation
krb5kdc: starting...
W20260927 10:47:50.286744 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:47:50 dist-test-slave-dzhw krb5kdc[20734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:47:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:47:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:47:50.413960 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34001
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:50.525527 20747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:50.525864 20747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:50.525954 20747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:50.530041 20747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:50.530143 20747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:50.530180 20747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:50.530215 20747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:50.530254 20747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:50.535758 20747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34001
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:50.536917 20747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:50.537962 20747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:50.546593 20753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:50.546655 20755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:50.546758 20747 server_base.cc:1034] running on GCE node
W20260927 10:47:50.546618 20752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:50.547506 20747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:50.548661 20747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:50.549851 20747 hybrid_clock.cc:648] HybridClock initialized: now 1790506070549835 us; error 35 us; skew 500 ppm
Sep 27 10:47:50 dist-test-slave-dzhw krb5kdc[20734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:50.552620 20747 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:47:50.553557 20747 webserver.cc:460] Webserver started at http://127.0.0.1:33279/ using document root <none> and password file <none>
I20260927 10:47:50.554141 20747 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:50.554229 20747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:50.554507 20747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:50.556596 20747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "82e6468f91d7447dac9feab469cda3e4"
format_stamp: "Formatted at 2026-09-27 10:47:50 on dist-test-slave-dzhw"
server_key: "d36e4af9b4b251e7f54db37c6b5ccc19"
server_key_iv: "8814e3b19f250f853668e32f9c98d5f7"
server_key_version: "encryptionkey@0"
I20260927 10:47:50.557163 20747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "82e6468f91d7447dac9feab469cda3e4"
format_stamp: "Formatted at 2026-09-27 10:47:50 on dist-test-slave-dzhw"
server_key: "d36e4af9b4b251e7f54db37c6b5ccc19"
server_key_iv: "8814e3b19f250f853668e32f9c98d5f7"
server_key_version: "encryptionkey@0"
I20260927 10:47:50.561002 20747 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 10:47:50.563484 20762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:50.564750 20747 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:47:50.564903 20747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "82e6468f91d7447dac9feab469cda3e4"
format_stamp: "Formatted at 2026-09-27 10:47:50 on dist-test-slave-dzhw"
server_key: "d36e4af9b4b251e7f54db37c6b5ccc19"
server_key_iv: "8814e3b19f250f853668e32f9c98d5f7"
server_key_version: "encryptionkey@0"
I20260927 10:47:50.565021 20747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:50.586383 20747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:50.589677 20747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:50.589908 20747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:50.597050 20747 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34001
I20260927 10:47:50.597074 20814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34001 every 8 connection(s)
I20260927 10:47:50.598059 20747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:47:50.600647 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 20747
I20260927 10:47:50.600826 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:47:50.601018 20815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:50.601125 392 external_mini_cluster.cc:1428] Setting key f94460d39e987bcddf6799564176e633
I20260927 10:47:50.607592 20815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e6468f91d7447dac9feab469cda3e4: Bootstrap starting.
I20260927 10:47:50.610090 20815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82e6468f91d7447dac9feab469cda3e4: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:50.610255 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:50.602573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33415 (local address 127.0.0.1:34001)
0927 10:47:50.603058 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:50.603073 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:50.603110 (+ 37us) server_negotiation.cc:407] Connection header received
0927 10:47:50.603769 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:50.603786 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:50.604129 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:50.604478 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:50.604877 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:50.605870 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:50.606850 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:50.607345 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:50.607520 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:50.608545 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:50.608551 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:50.608564 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:47:50.608976 (+ 412us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:50.609033 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:47:50.609056 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:50.609198 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:50.609368 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 10:47:50.609511 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":150,"threads_started":1}
I20260927 10:47:50.610973 20815 log.cc:824] T 00000000000000000000000000000000 P 82e6468f91d7447dac9feab469cda3e4: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:50.613148 20815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e6468f91d7447dac9feab469cda3e4: No bootstrap required, opened a new log
I20260927 10:47:50.613281 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:47:50.617548 20815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82e6468f91d7447dac9feab469cda3e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e6468f91d7447dac9feab469cda3e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34001 } }
I20260927 10:47:50.617854 20815 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82e6468f91d7447dac9feab469cda3e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:50.617919 20815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82e6468f91d7447dac9feab469cda3e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82e6468f91d7447dac9feab469cda3e4, State: Initialized, Role: FOLLOWER
I20260927 10:47:50.618448 20815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82e6468f91d7447dac9feab469cda3e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e6468f91d7447dac9feab469cda3e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34001 } }
I20260927 10:47:50.618628 20815 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82e6468f91d7447dac9feab469cda3e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:47:50.618711 20815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82e6468f91d7447dac9feab469cda3e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:47:50.618840 20815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82e6468f91d7447dac9feab469cda3e4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:50.619345 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:50.615295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54523 (local address 127.0.0.1:34001)
0927 10:47:50.615457 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:50.615463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:50.615516 (+ 53us) server_negotiation.cc:407] Connection header received
0927 10:47:50.615635 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:50.615638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:50.615686 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:50.615773 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:50.615999 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:50.616771 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:50.617574 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:50.617889 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:50.618005 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:50.618913 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:50.618916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:50.618919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:47:50.618932 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:50.618963 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:47:50.618975 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:50.619055 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:50.619170 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 10:47:50.619203 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:47:50.620057 20815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82e6468f91d7447dac9feab469cda3e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e6468f91d7447dac9feab469cda3e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34001 } }
I20260927 10:47:50.620437 20815 leader_election.cc:302] T 00000000000000000000000000000000 P 82e6468f91d7447dac9feab469cda3e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82e6468f91d7447dac9feab469cda3e4; no voters:
I20260927 10:47:50.620709 20815 leader_election.cc:288] T 00000000000000000000000000000000 P 82e6468f91d7447dac9feab469cda3e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:47:50.620814 20825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82e6468f91d7447dac9feab469cda3e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:50.621070 20825 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82e6468f91d7447dac9feab469cda3e4 [term 1 LEADER]: Becoming Leader. State: Replica: 82e6468f91d7447dac9feab469cda3e4, State: Running, Role: LEADER
I20260927 10:47:50.621390 20825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82e6468f91d7447dac9feab469cda3e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e6468f91d7447dac9feab469cda3e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34001 } }
I20260927 10:47:50.621989 20815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82e6468f91d7447dac9feab469cda3e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:47:50.623075 20827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e6468f91d7447dac9feab469cda3e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82e6468f91d7447dac9feab469cda3e4. Latest consensus state: current_term: 1 leader_uuid: "82e6468f91d7447dac9feab469cda3e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e6468f91d7447dac9feab469cda3e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34001 } } }
I20260927 10:47:50.623191 20827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e6468f91d7447dac9feab469cda3e4 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:50.623282 20826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e6468f91d7447dac9feab469cda3e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82e6468f91d7447dac9feab469cda3e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e6468f91d7447dac9feab469cda3e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34001 } } }
I20260927 10:47:50.623381 20826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e6468f91d7447dac9feab469cda3e4 [sys.catalog]: This master's current role is: LEADER
W20260927 10:47:50.626636 20840 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82e6468f91d7447dac9feab469cda3e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:47:50.626727 20840 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:47:50.626798 20834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:47:50.627468 20834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:47:50.633330 20834 catalog_manager.cc:1347] Generated new cluster ID: 9f612b19d92044b78eaec2bd7eb658b4
I20260927 10:47:50.633416 20834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:47:50.666728 20834 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:47:50.668128 20834 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:47:50.674844 20834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82e6468f91d7447dac9feab469cda3e4: Generated new TSK 0
I20260927 10:47:50.675709 20834 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:47:50.689437 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 20747
W20260927 10:47:50.695062 20732 connection.cc:535] client connection to 127.0.0.1:34001 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34001): Connection reset by peer (error 104)
2026-09-27T10:47:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1438 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:47:50 dist-test-slave-dzhw krb5kdc[20846](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:47:50 dist-test-slave-dzhw krb5kdc[20846](info): set up 2 sockets
Sep 27 10:47:50 dist-test-slave-dzhw krb5kdc[20846](info): commencing operation
krb5kdc: starting...
W20260927 10:47:51.730656 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:47:51 dist-test-slave-dzhw krb5kdc[20846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:47:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:47:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:47:51.851343 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33173
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:51.960098 20859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:51.960435 20859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:51.960511 20859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:51.964519 20859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:51.964624 20859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:51.964663 20859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:51.964710 20859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:51.964745 20859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:51.970108 20859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33173
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:51.971235 20859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:51.972249 20859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:51.980690 20865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:51.980688 20864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:51.980686 20867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:51.980996 20859 server_base.cc:1034] running on GCE node
I20260927 10:47:51.981571 20859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:51.982719 20859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:51.983904 20859 hybrid_clock.cc:648] HybridClock initialized: now 1790506071983882 us; error 33 us; skew 500 ppm
Sep 27 10:47:51 dist-test-slave-dzhw krb5kdc[20846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:51.986763 20859 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:47:51.987685 20859 webserver.cc:460] Webserver started at http://127.0.0.1:34695/ using document root <none> and password file <none>
I20260927 10:47:51.988256 20859 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:51.988324 20859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:51.988564 20859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:51.990752 20859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "635341ce874b48eebaa27bf4dbfbdc69"
format_stamp: "Formatted at 2026-09-27 10:47:51 on dist-test-slave-dzhw"
server_key: "8e8a054a4ca8266e57349d4ce4328f6a"
server_key_iv: "41083912777645827e3e0d33f0afa4c9"
server_key_version: "encryptionkey@0"
I20260927 10:47:51.991293 20859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "635341ce874b48eebaa27bf4dbfbdc69"
format_stamp: "Formatted at 2026-09-27 10:47:51 on dist-test-slave-dzhw"
server_key: "8e8a054a4ca8266e57349d4ce4328f6a"
server_key_iv: "41083912777645827e3e0d33f0afa4c9"
server_key_version: "encryptionkey@0"
I20260927 10:47:51.994916 20859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:47:51.997224 20874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:51.998517 20859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:47:51.998649 20859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "635341ce874b48eebaa27bf4dbfbdc69"
format_stamp: "Formatted at 2026-09-27 10:47:51 on dist-test-slave-dzhw"
server_key: "8e8a054a4ca8266e57349d4ce4328f6a"
server_key_iv: "41083912777645827e3e0d33f0afa4c9"
server_key_version: "encryptionkey@0"
I20260927 10:47:51.998759 20859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:52.003330 20859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:52.006623 20859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:52.006837 20859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:52.013748 20859 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33173
I20260927 10:47:52.013774 20926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33173 every 8 connection(s)
I20260927 10:47:52.014746 20859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:47:52.017133 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 20859
I20260927 10:47:52.017295 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:47:52.017520 20927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:52.017606 392 external_mini_cluster.cc:1428] Setting key a4a02f6066820c447d1eb766ce18a540
I20260927 10:47:52.022828 20930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:52.019002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53857 (local address 127.0.0.1:33173)
0927 10:47:52.019461 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:52.019476 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:52.019510 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:47:52.020190 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:52.020207 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:52.020512 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:52.020857 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:52.021638 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:52.021643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:52.021654 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:47:52.021664 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:52.021713 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:47:52.021731 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:52.021869 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:52.021943 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 10:47:52.022097 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":143,"threads_started":1}
I20260927 10:47:52.024132 20927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635341ce874b48eebaa27bf4dbfbdc69: Bootstrap starting.
I20260927 10:47:52.025563 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:47:52.026592 20927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 635341ce874b48eebaa27bf4dbfbdc69: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:52.027319 20927 log.cc:824] T 00000000000000000000000000000000 P 635341ce874b48eebaa27bf4dbfbdc69: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:52.029268 20927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635341ce874b48eebaa27bf4dbfbdc69: No bootstrap required, opened a new log
I20260927 10:47:52.029404 20930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:52.027632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55989 (local address 127.0.0.1:33173)
0927 10:47:52.027770 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:52.027775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:52.027979 (+ 204us) server_negotiation.cc:407] Connection header received
0927 10:47:52.028098 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:52.028102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:52.028135 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:52.028229 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:52.028943 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:52.028946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:52.028949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:47:52.028959 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:52.028985 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:47:52.029003 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:52.029090 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:52.029174 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 10:47:52.029211 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:47:52.033111 20927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 635341ce874b48eebaa27bf4dbfbdc69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635341ce874b48eebaa27bf4dbfbdc69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33173 } }
I20260927 10:47:52.033380 20927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 635341ce874b48eebaa27bf4dbfbdc69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:52.033444 20927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 635341ce874b48eebaa27bf4dbfbdc69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 635341ce874b48eebaa27bf4dbfbdc69, State: Initialized, Role: FOLLOWER
I20260927 10:47:52.033867 20927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 635341ce874b48eebaa27bf4dbfbdc69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635341ce874b48eebaa27bf4dbfbdc69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33173 } }
I20260927 10:47:52.034015 20927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 635341ce874b48eebaa27bf4dbfbdc69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:47:52.034082 20927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 635341ce874b48eebaa27bf4dbfbdc69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:47:52.034183 20927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 635341ce874b48eebaa27bf4dbfbdc69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:52.035208 20927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 635341ce874b48eebaa27bf4dbfbdc69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635341ce874b48eebaa27bf4dbfbdc69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33173 } }
I20260927 10:47:52.035578 20927 leader_election.cc:302] T 00000000000000000000000000000000 P 635341ce874b48eebaa27bf4dbfbdc69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 635341ce874b48eebaa27bf4dbfbdc69; no voters:
I20260927 10:47:52.035946 20927 leader_election.cc:288] T 00000000000000000000000000000000 P 635341ce874b48eebaa27bf4dbfbdc69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:47:52.036017 20937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 635341ce874b48eebaa27bf4dbfbdc69 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:52.036226 20937 raft_consensus.cc:686] T 00000000000000000000000000000000 P 635341ce874b48eebaa27bf4dbfbdc69 [term 1 LEADER]: Becoming Leader. State: Replica: 635341ce874b48eebaa27bf4dbfbdc69, State: Running, Role: LEADER
I20260927 10:47:52.036561 20937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 635341ce874b48eebaa27bf4dbfbdc69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635341ce874b48eebaa27bf4dbfbdc69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33173 } }
I20260927 10:47:52.037297 20927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 635341ce874b48eebaa27bf4dbfbdc69 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:47:52.038337 20939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635341ce874b48eebaa27bf4dbfbdc69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 635341ce874b48eebaa27bf4dbfbdc69. Latest consensus state: current_term: 1 leader_uuid: "635341ce874b48eebaa27bf4dbfbdc69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635341ce874b48eebaa27bf4dbfbdc69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33173 } } }
I20260927 10:47:52.038506 20939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635341ce874b48eebaa27bf4dbfbdc69 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:52.038782 20938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635341ce874b48eebaa27bf4dbfbdc69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "635341ce874b48eebaa27bf4dbfbdc69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635341ce874b48eebaa27bf4dbfbdc69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33173 } } }
I20260927 10:47:52.038908 20946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:47:52.038923 20938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635341ce874b48eebaa27bf4dbfbdc69 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:52.041591 20946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:47:52.046826 20946 catalog_manager.cc:1347] Generated new cluster ID: 1069f69a56e8492e8083a5dade72b44f
I20260927 10:47:52.046911 20946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:47:52.074028 20946 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:47:52.074375 20946 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:47:52.082773 20946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 635341ce874b48eebaa27bf4dbfbdc69: Generated new TSK 0
I20260927 10:47:52.083634 20946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:47:52.090916 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 20859
2026-09-27T10:47:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:47:52 dist-test-slave-dzhw krb5kdc[20958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:47:52 dist-test-slave-dzhw krb5kdc[20958](info): set up 2 sockets
Sep 27 10:47:52 dist-test-slave-dzhw krb5kdc[20958](info): commencing operation
krb5kdc: starting...
W20260927 10:47:53.151595 392 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:47:53 dist-test-slave-dzhw krb5kdc[20958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:47:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:47:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:47:53.292060 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37845
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:53.424407 20971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:53.424816 20971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:53.424919 20971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:53.429384 20971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:53.429487 20971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:53.429528 20971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:53.429566 20971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:53.429605 20971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:53.435981 20971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:53.437853 20971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:53.439391 20971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:53.449957 20976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:53.450019 20971 server_base.cc:1034] running on GCE node
W20260927 10:47:53.450171 20977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:53.450019 20979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:53.451068 20971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:53.452582 20971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:53.453821 20971 hybrid_clock.cc:648] HybridClock initialized: now 1790506073453786 us; error 53 us; skew 500 ppm
Sep 27 10:47:53 dist-test-slave-dzhw krb5kdc[20958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:53.465454 20971 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:47:53.466889 20971 webserver.cc:460] Webserver started at http://127.0.0.1:45753/ using document root <none> and password file <none>
I20260927 10:47:53.467666 20971 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:53.467765 20971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:53.468041 20971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:53.470734 20971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "5e7f7a2c77af4aaebc0ec32f59f63572"
format_stamp: "Formatted at 2026-09-27 10:47:53 on dist-test-slave-dzhw"
server_key: "49bcdea89586839f9f52e70c750ea9ad"
server_key_iv: "d45a1850781bbd7887f6da736f57fabe"
server_key_version: "encryptionkey@0"
I20260927 10:47:53.471421 20971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "5e7f7a2c77af4aaebc0ec32f59f63572"
format_stamp: "Formatted at 2026-09-27 10:47:53 on dist-test-slave-dzhw"
server_key: "49bcdea89586839f9f52e70c750ea9ad"
server_key_iv: "d45a1850781bbd7887f6da736f57fabe"
server_key_version: "encryptionkey@0"
I20260927 10:47:53.476060 20971 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:47:53.479267 20986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:53.480942 20971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:47:53.481130 20971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "5e7f7a2c77af4aaebc0ec32f59f63572"
format_stamp: "Formatted at 2026-09-27 10:47:53 on dist-test-slave-dzhw"
server_key: "49bcdea89586839f9f52e70c750ea9ad"
server_key_iv: "d45a1850781bbd7887f6da736f57fabe"
server_key_version: "encryptionkey@0"
I20260927 10:47:53.481280 20971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:53.500348 20971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:53.504668 20971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:53.504932 20971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:53.514278 20971 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37845
I20260927 10:47:53.514323 21038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37845 every 8 connection(s)
I20260927 10:47:53.515812 20971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:47:53.519944 21039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:53.521340 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 20971
I20260927 10:47:53.521517 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:47:53.521926 392 external_mini_cluster.cc:1428] Setting key 6396f482bfaca9b5b578cd265f248387
I20260927 10:47:53.528695 21039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7f7a2c77af4aaebc0ec32f59f63572: Bootstrap starting.
I20260927 10:47:53.532141 21039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e7f7a2c77af4aaebc0ec32f59f63572: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:53.533113 21039 log.cc:824] T 00000000000000000000000000000000 P 5e7f7a2c77af4aaebc0ec32f59f63572: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:47:53 dist-test-slave-dzhw krb5kdc[20958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 10:47:53.536265 21039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7f7a2c77af4aaebc0ec32f59f63572: No bootstrap required, opened a new log
I20260927 10:47:53.540808 21042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:53.523914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43997 (local address 127.0.0.1:37845)
0927 10:47:53.524469 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:53.524493 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:53.524546 (+ 53us) server_negotiation.cc:407] Connection header received
0927 10:47:53.525517 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:53.525547 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:53.526090 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:53.526596 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:53.527447 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:53.529094 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:53.530181 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:53.531007 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:53.531199 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:53.534669 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:53.534676 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:53.534697 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:53.534730 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:53.536878 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:53.537595 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:53.537603 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:53.537612 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:53.537767 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:53.538158 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:53.538162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:53.538163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:53.538798 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:53.538997 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:53.539395 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 10:47:53.539796 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":162,"threads_started":1}
I20260927 10:47:53.541419 21039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e7f7a2c77af4aaebc0ec32f59f63572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7f7a2c77af4aaebc0ec32f59f63572" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37845 } }
I20260927 10:47:53.541751 21039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e7f7a2c77af4aaebc0ec32f59f63572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:53.541838 21039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e7f7a2c77af4aaebc0ec32f59f63572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7f7a2c77af4aaebc0ec32f59f63572, State: Initialized, Role: FOLLOWER
I20260927 10:47:53.542369 21039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e7f7a2c77af4aaebc0ec32f59f63572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7f7a2c77af4aaebc0ec32f59f63572" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37845 } }
I20260927 10:47:53.542620 21039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e7f7a2c77af4aaebc0ec32f59f63572 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:47:53.542713 21039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e7f7a2c77af4aaebc0ec32f59f63572 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:47:53.542843 21039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e7f7a2c77af4aaebc0ec32f59f63572 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:53.544310 21039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e7f7a2c77af4aaebc0ec32f59f63572 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7f7a2c77af4aaebc0ec32f59f63572" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37845 } }
I20260927 10:47:53.544716 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:47:53.544824 21039 leader_election.cc:302] T 00000000000000000000000000000000 P 5e7f7a2c77af4aaebc0ec32f59f63572 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e7f7a2c77af4aaebc0ec32f59f63572; no voters:
I20260927 10:47:53.545275 21039 leader_election.cc:288] T 00000000000000000000000000000000 P 5e7f7a2c77af4aaebc0ec32f59f63572 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:47:53.545416 21044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e7f7a2c77af4aaebc0ec32f59f63572 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:53.545691 21044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e7f7a2c77af4aaebc0ec32f59f63572 [term 1 LEADER]: Becoming Leader. State: Replica: 5e7f7a2c77af4aaebc0ec32f59f63572, State: Running, Role: LEADER
I20260927 10:47:53.546145 21044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e7f7a2c77af4aaebc0ec32f59f63572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7f7a2c77af4aaebc0ec32f59f63572" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37845 } }
I20260927 10:47:53.547108 21039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e7f7a2c77af4aaebc0ec32f59f63572 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:47:53.548226 21046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7f7a2c77af4aaebc0ec32f59f63572 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e7f7a2c77af4aaebc0ec32f59f63572. Latest consensus state: current_term: 1 leader_uuid: "5e7f7a2c77af4aaebc0ec32f59f63572" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7f7a2c77af4aaebc0ec32f59f63572" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37845 } } }
I20260927 10:47:53.548399 21046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7f7a2c77af4aaebc0ec32f59f63572 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:53.548839 21045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7f7a2c77af4aaebc0ec32f59f63572 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e7f7a2c77af4aaebc0ec32f59f63572" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7f7a2c77af4aaebc0ec32f59f63572" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37845 } } }
I20260927 10:47:53.548918 21058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:47:53.548934 21045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7f7a2c77af4aaebc0ec32f59f63572 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:53.553062 21058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:47:53.559321 21042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:53.549171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33843 (local address 127.0.0.1:37845)
0927 10:47:53.549282 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:53.549287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:53.549304 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:47:53.549352 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:53.549355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:53.549410 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:53.549510 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:53.550641 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:53.551450 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:53.553070 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:53.553428 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:53.553541 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:53.555685 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:53.555691 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:53.555694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:53.555738 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:53.557410 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:53.557895 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:53.557899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:53.557901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:53.557994 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:53.558286 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:53.558292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:53.558297 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:53.558585 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:53.558740 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:53.558986 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 10:47:53.559128 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 10:47:53.561390 21058 catalog_manager.cc:1347] Generated new cluster ID: d1397d49fff54298bebaa36bad2f8260
I20260927 10:47:53.561491 21058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:47:53.574359 21058 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:47:53.575750 21058 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:47:53.583427 21058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e7f7a2c77af4aaebc0ec32f59f63572: Generated new TSK 0
I20260927 10:47:53.584728 21058 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:47:53.597141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 20971
2026-09-27T10:47:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1507 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:47:53 dist-test-slave-dzhw krb5kdc[21070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:47:53 dist-test-slave-dzhw krb5kdc[21070](info): set up 2 sockets
Sep 27 10:47:53 dist-test-slave-dzhw krb5kdc[21070](info): commencing operation
krb5kdc: starting...
W20260927 10:47:54.655157 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:47:54 dist-test-slave-dzhw krb5kdc[21070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:47:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:47:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:47:54.794304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35761
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:54.927475 21083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:54.927862 21083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:54.927940 21083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:54.932263 21083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:54.932366 21083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:54.932403 21083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:54.932461 21083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:54.932507 21083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:54.938345 21083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:54.939785 21083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:54.941068 21083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:54.950855 21088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:54.950907 21091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:54.950848 21089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:54.951326 21083 server_base.cc:1034] running on GCE node
I20260927 10:47:54.951977 21083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:54.953334 21083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:54.954493 21083 hybrid_clock.cc:648] HybridClock initialized: now 1790506074954473 us; error 28 us; skew 500 ppm
Sep 27 10:47:54 dist-test-slave-dzhw krb5kdc[21070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:54.957618 21083 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:47:54.958743 21083 webserver.cc:460] Webserver started at http://127.0.0.1:35357/ using document root <none> and password file <none>
I20260927 10:47:54.959421 21083 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:54.959496 21083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:54.959770 21083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:54.962253 21083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "a0b7d77f0547433a84de84f4997f3ff7"
format_stamp: "Formatted at 2026-09-27 10:47:54 on dist-test-slave-dzhw"
server_key: "1bf4bba9681069bc15e1402d9f4074d8"
server_key_iv: "cb223ae388e21b4aa41adf13e4495866"
server_key_version: "encryptionkey@0"
I20260927 10:47:54.962963 21083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "a0b7d77f0547433a84de84f4997f3ff7"
format_stamp: "Formatted at 2026-09-27 10:47:54 on dist-test-slave-dzhw"
server_key: "1bf4bba9681069bc15e1402d9f4074d8"
server_key_iv: "cb223ae388e21b4aa41adf13e4495866"
server_key_version: "encryptionkey@0"
I20260927 10:47:54.967399 21083 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:47:54.970543 21098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:54.972230 21083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:47:54.972427 21083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "a0b7d77f0547433a84de84f4997f3ff7"
format_stamp: "Formatted at 2026-09-27 10:47:54 on dist-test-slave-dzhw"
server_key: "1bf4bba9681069bc15e1402d9f4074d8"
server_key_iv: "cb223ae388e21b4aa41adf13e4495866"
server_key_version: "encryptionkey@0"
I20260927 10:47:54.972587 21083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:54.995366 21083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:54.999339 21083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:54.999601 21083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:55.008535 21083 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35761
I20260927 10:47:55.008565 21150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35761 every 8 connection(s)
I20260927 10:47:55.009776 21083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:47:55.013288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 21083
I20260927 10:47:55.013446 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:47:55.013790 392 external_mini_cluster.cc:1428] Setting key 31de9183423a43963fcb6a07b56a5ef2
I20260927 10:47:55.014154 21151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:55.022130 21151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0b7d77f0547433a84de84f4997f3ff7: Bootstrap starting.
I20260927 10:47:55.023837 21154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:55.015423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42517 (local address 127.0.0.1:35761)
0927 10:47:55.015912 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:55.015936 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:55.015978 (+ 42us) server_negotiation.cc:407] Connection header received
0927 10:47:55.016738 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:55.016759 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:55.017165 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:55.017500 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:55.017891 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:55.019030 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:55.020231 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:55.020723 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:55.020845 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:55.021825 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:55.021830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:55.021844 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:47:55.022327 (+ 483us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:55.022401 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:47:55.022479 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:55.022566 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:55.022705 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 10:47:55.022966 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":179,"threads_started":1}
I20260927 10:47:55.024971 21151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0b7d77f0547433a84de84f4997f3ff7: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:55.025892 21151 log.cc:824] T 00000000000000000000000000000000 P a0b7d77f0547433a84de84f4997f3ff7: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:55.027136 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:47:55.028322 21151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0b7d77f0547433a84de84f4997f3ff7: No bootstrap required, opened a new log
I20260927 10:47:55.033483 21151 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0b7d77f0547433a84de84f4997f3ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b7d77f0547433a84de84f4997f3ff7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35761 } }
I20260927 10:47:55.033869 21151 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0b7d77f0547433a84de84f4997f3ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:55.033860 21154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:55.029424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51221 (local address 127.0.0.1:35761)
0927 10:47:55.029567 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:55.029573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:55.029756 (+ 183us) server_negotiation.cc:407] Connection header received
0927 10:47:55.029895 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:55.029897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:55.029938 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:55.030006 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:55.030309 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:55.030938 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:55.032116 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:55.032356 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:55.032440 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:55.033348 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:55.033351 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:55.033354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:47:55.033366 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:55.033391 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:47:55.033401 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:55.033454 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:55.033607 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 10:47:55.033676 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:47:55.033944 21151 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0b7d77f0547433a84de84f4997f3ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0b7d77f0547433a84de84f4997f3ff7, State: Initialized, Role: FOLLOWER
I20260927 10:47:55.034533 21151 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0b7d77f0547433a84de84f4997f3ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b7d77f0547433a84de84f4997f3ff7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35761 } }
I20260927 10:47:55.034711 21151 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0b7d77f0547433a84de84f4997f3ff7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:47:55.034785 21151 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0b7d77f0547433a84de84f4997f3ff7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:47:55.034894 21151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0b7d77f0547433a84de84f4997f3ff7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:55.035998 21151 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0b7d77f0547433a84de84f4997f3ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b7d77f0547433a84de84f4997f3ff7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35761 } }
I20260927 10:47:55.036410 21151 leader_election.cc:302] T 00000000000000000000000000000000 P a0b7d77f0547433a84de84f4997f3ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0b7d77f0547433a84de84f4997f3ff7; no voters:
I20260927 10:47:55.037029 21151 leader_election.cc:288] T 00000000000000000000000000000000 P a0b7d77f0547433a84de84f4997f3ff7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:47:55.037139 21161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0b7d77f0547433a84de84f4997f3ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:55.037464 21161 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0b7d77f0547433a84de84f4997f3ff7 [term 1 LEADER]: Becoming Leader. State: Replica: a0b7d77f0547433a84de84f4997f3ff7, State: Running, Role: LEADER
I20260927 10:47:55.037966 21161 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0b7d77f0547433a84de84f4997f3ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b7d77f0547433a84de84f4997f3ff7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35761 } }
I20260927 10:47:55.038733 21151 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0b7d77f0547433a84de84f4997f3ff7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:47:55.039888 21163 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0b7d77f0547433a84de84f4997f3ff7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0b7d77f0547433a84de84f4997f3ff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b7d77f0547433a84de84f4997f3ff7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35761 } } }
I20260927 10:47:55.040021 21163 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0b7d77f0547433a84de84f4997f3ff7 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:55.040380 21162 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0b7d77f0547433a84de84f4997f3ff7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0b7d77f0547433a84de84f4997f3ff7. Latest consensus state: current_term: 1 leader_uuid: "a0b7d77f0547433a84de84f4997f3ff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b7d77f0547433a84de84f4997f3ff7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35761 } } }
I20260927 10:47:55.040473 21162 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0b7d77f0547433a84de84f4997f3ff7 [sys.catalog]: This master's current role is: LEADER
W20260927 10:47:55.044495 21176 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0b7d77f0547433a84de84f4997f3ff7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:47:55.044589 21176 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:47:55.044667 21170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:47:55.045406 21170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:47:55.052403 21170 catalog_manager.cc:1347] Generated new cluster ID: dcaf0b4ea00c492a827488513b298862
I20260927 10:47:55.052613 21170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:47:55.067466 21170 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:47:55.068647 21170 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:47:55.073441 21170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0b7d77f0547433a84de84f4997f3ff7: Generated new TSK 0
I20260927 10:47:55.074595 21170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:47:55.084085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 21083
2026-09-27T10:47:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1486 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:47:55 dist-test-slave-dzhw krb5kdc[21182](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:47:55 dist-test-slave-dzhw krb5kdc[21182](info): set up 2 sockets
Sep 27 10:47:55 dist-test-slave-dzhw krb5kdc[21182](info): commencing operation
krb5kdc: starting...
W20260927 10:47:56.147131 392 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:47:56 dist-test-slave-dzhw krb5kdc[21182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:47:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:47:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:47:56.294584 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44461
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:56.437824 21195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:56.438231 21195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:56.438318 21195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:56.442881 21195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:56.442991 21195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:56.443034 21195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:56.443073 21195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:56.443111 21195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:56.449815 21195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:56.451568 21195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:56.452963 21195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:56.463650 21200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:56.463852 21195 server_base.cc:1034] running on GCE node
W20260927 10:47:56.463740 21201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:56.463641 21203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:56.464777 21195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:56.466264 21195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:56.467495 21195 hybrid_clock.cc:648] HybridClock initialized: now 1790506076467457 us; error 56 us; skew 500 ppm
Sep 27 10:47:56 dist-test-slave-dzhw krb5kdc[21182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:56.470872 21195 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:47:56.471997 21195 webserver.cc:460] Webserver started at http://127.0.0.1:38515/ using document root <none> and password file <none>
I20260927 10:47:56.472698 21195 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:56.472769 21195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:56.472972 21195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:56.475597 21195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "4aa8360b9d7d4c6aa91f49386b4322ad"
format_stamp: "Formatted at 2026-09-27 10:47:56 on dist-test-slave-dzhw"
server_key: "eceffb853edb5fbcc073783f47b4661a"
server_key_iv: "c19a5e0f5fc1f23ddcb77fe4faf84312"
server_key_version: "encryptionkey@0"
I20260927 10:47:56.476210 21195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "4aa8360b9d7d4c6aa91f49386b4322ad"
format_stamp: "Formatted at 2026-09-27 10:47:56 on dist-test-slave-dzhw"
server_key: "eceffb853edb5fbcc073783f47b4661a"
server_key_iv: "c19a5e0f5fc1f23ddcb77fe4faf84312"
server_key_version: "encryptionkey@0"
I20260927 10:47:56.480719 21195 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:47:56.483635 21210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:56.485112 21195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:47:56.485297 21195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "4aa8360b9d7d4c6aa91f49386b4322ad"
format_stamp: "Formatted at 2026-09-27 10:47:56 on dist-test-slave-dzhw"
server_key: "eceffb853edb5fbcc073783f47b4661a"
server_key_iv: "c19a5e0f5fc1f23ddcb77fe4faf84312"
server_key_version: "encryptionkey@0"
I20260927 10:47:56.485435 21195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:56.491019 21195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:56.495249 21195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:56.495530 21195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:56.504750 21195 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44461
I20260927 10:47:56.504776 21262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44461 every 8 connection(s)
I20260927 10:47:56.506116 21195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:47:56.510025 21263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:56.513610 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 21195
I20260927 10:47:56.513808 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:47:56.514191 392 external_mini_cluster.cc:1428] Setting key c6c5d1af14f17596ea5952156d9e4c30
I20260927 10:47:56.518278 21263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aa8360b9d7d4c6aa91f49386b4322ad: Bootstrap starting.
I20260927 10:47:56.520644 21266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:56.516123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54575 (local address 127.0.0.1:44461)
0927 10:47:56.516707 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:56.516726 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:56.516778 (+ 52us) server_negotiation.cc:407] Connection header received
0927 10:47:56.517517 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:56.517538 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:56.517904 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:56.518336 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:56.519258 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:56.519262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:56.519277 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:47:56.519287 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:56.519348 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:47:56.519370 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:56.519526 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:56.519608 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 10:47:56.519790 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":181,"threads_started":1}
I20260927 10:47:56.521646 21263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4aa8360b9d7d4c6aa91f49386b4322ad: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:56.522671 21263 log.cc:824] T 00000000000000000000000000000000 P 4aa8360b9d7d4c6aa91f49386b4322ad: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:56.524145 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:47:56.525213 21263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aa8360b9d7d4c6aa91f49386b4322ad: No bootstrap required, opened a new log
I20260927 10:47:56.528697 21266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:56.526690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44929 (local address 127.0.0.1:44461)
0927 10:47:56.526840 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:56.526845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:56.527091 (+ 246us) server_negotiation.cc:407] Connection header received
0927 10:47:56.527222 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:56.527224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:56.527260 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:56.527329 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:56.528168 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:56.528171 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:56.528173 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:47:56.528184 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:56.528212 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:47:56.528225 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:56.528319 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:56.528438 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 10:47:56.528474 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:47:56.530558 21263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4aa8360b9d7d4c6aa91f49386b4322ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa8360b9d7d4c6aa91f49386b4322ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44461 } }
I20260927 10:47:56.530897 21263 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4aa8360b9d7d4c6aa91f49386b4322ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:56.530983 21263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4aa8360b9d7d4c6aa91f49386b4322ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aa8360b9d7d4c6aa91f49386b4322ad, State: Initialized, Role: FOLLOWER
I20260927 10:47:56.531546 21263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4aa8360b9d7d4c6aa91f49386b4322ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa8360b9d7d4c6aa91f49386b4322ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44461 } }
I20260927 10:47:56.531734 21263 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4aa8360b9d7d4c6aa91f49386b4322ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:47:56.531837 21263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4aa8360b9d7d4c6aa91f49386b4322ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:47:56.531955 21263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4aa8360b9d7d4c6aa91f49386b4322ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:56.533217 21263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4aa8360b9d7d4c6aa91f49386b4322ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa8360b9d7d4c6aa91f49386b4322ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44461 } }
I20260927 10:47:56.533701 21263 leader_election.cc:302] T 00000000000000000000000000000000 P 4aa8360b9d7d4c6aa91f49386b4322ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4aa8360b9d7d4c6aa91f49386b4322ad; no voters:
I20260927 10:47:56.534157 21263 leader_election.cc:288] T 00000000000000000000000000000000 P 4aa8360b9d7d4c6aa91f49386b4322ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:47:56.534384 21273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4aa8360b9d7d4c6aa91f49386b4322ad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:56.534749 21273 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4aa8360b9d7d4c6aa91f49386b4322ad [term 1 LEADER]: Becoming Leader. State: Replica: 4aa8360b9d7d4c6aa91f49386b4322ad, State: Running, Role: LEADER
I20260927 10:47:56.535220 21273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4aa8360b9d7d4c6aa91f49386b4322ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa8360b9d7d4c6aa91f49386b4322ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44461 } }
I20260927 10:47:56.535957 21263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4aa8360b9d7d4c6aa91f49386b4322ad [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:47:56.537696 21275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aa8360b9d7d4c6aa91f49386b4322ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4aa8360b9d7d4c6aa91f49386b4322ad. Latest consensus state: current_term: 1 leader_uuid: "4aa8360b9d7d4c6aa91f49386b4322ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa8360b9d7d4c6aa91f49386b4322ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44461 } } }
I20260927 10:47:56.537695 21274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aa8360b9d7d4c6aa91f49386b4322ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4aa8360b9d7d4c6aa91f49386b4322ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa8360b9d7d4c6aa91f49386b4322ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44461 } } }
I20260927 10:47:56.537875 21275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aa8360b9d7d4c6aa91f49386b4322ad [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:56.537875 21274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aa8360b9d7d4c6aa91f49386b4322ad [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:56.538369 21285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:47:56.542199 21285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:47:56.549501 21285 catalog_manager.cc:1347] Generated new cluster ID: e104b62df14449fc89768ed84c09680c
I20260927 10:47:56.549613 21285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:47:56.566864 21285 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:47:56.567322 21285 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:47:56.573908 21285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4aa8360b9d7d4c6aa91f49386b4322ad: Generated new TSK 0
I20260927 10:47:56.574960 21285 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:47:56.588364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 21195
2026-09-27T10:47:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1505 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:47:56 dist-test-slave-dzhw krb5kdc[21294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:47:56 dist-test-slave-dzhw krb5kdc[21294](info): set up 2 sockets
Sep 27 10:47:56 dist-test-slave-dzhw krb5kdc[21294](info): commencing operation
krb5kdc: starting...
W20260927 10:47:57.635099 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:47:57 dist-test-slave-dzhw krb5kdc[21294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:47:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:47:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:47:57.831017 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36489
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:57.962013 21307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:57.962368 21307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:57.962476 21307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:57.966715 21307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:57.966789 21307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:57.966809 21307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:57.966826 21307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:57.966842 21307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:57.972718 21307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:57.974112 21307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:57.975373 21307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:47:57.986320 21307 server_base.cc:1034] running on GCE node
W20260927 10:47:57.986320 21312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:57.986356 21315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:57.986333 21313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:57.987499 21307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:57.989107 21307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:57.990339 21307 hybrid_clock.cc:648] HybridClock initialized: now 1790506077990313 us; error 44 us; skew 500 ppm
Sep 27 10:47:57 dist-test-slave-dzhw krb5kdc[21294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:57.994059 21307 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:47:57.995364 21307 webserver.cc:460] Webserver started at http://127.0.0.1:33975/ using document root <none> and password file <none>
I20260927 10:47:57.996119 21307 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:57.996224 21307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:57.996516 21307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:57.999177 21307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "cf9c1bd605be4abe9b7f0569774fa627"
format_stamp: "Formatted at 2026-09-27 10:47:57 on dist-test-slave-dzhw"
server_key: "e6de4bf259606fc03fe1fb316ac0b766"
server_key_iv: "2a10d9454003ad6f35ca384eef7d3886"
server_key_version: "encryptionkey@0"
I20260927 10:47:57.999836 21307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "cf9c1bd605be4abe9b7f0569774fa627"
format_stamp: "Formatted at 2026-09-27 10:47:57 on dist-test-slave-dzhw"
server_key: "e6de4bf259606fc03fe1fb316ac0b766"
server_key_iv: "2a10d9454003ad6f35ca384eef7d3886"
server_key_version: "encryptionkey@0"
I20260927 10:47:58.004642 21307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 10:47:58.007877 21322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:58.009604 21307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:47:58.009811 21307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "cf9c1bd605be4abe9b7f0569774fa627"
format_stamp: "Formatted at 2026-09-27 10:47:57 on dist-test-slave-dzhw"
server_key: "e6de4bf259606fc03fe1fb316ac0b766"
server_key_iv: "2a10d9454003ad6f35ca384eef7d3886"
server_key_version: "encryptionkey@0"
I20260927 10:47:58.009974 21307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:58.046986 21307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:58.050801 21307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:58.051083 21307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:58.060298 21307 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36489
I20260927 10:47:58.060326 21374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36489 every 8 connection(s)
I20260927 10:47:58.061599 21307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:47:58.065393 21375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:58.070706 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 21307
I20260927 10:47:58.070931 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:47:58.071321 392 external_mini_cluster.cc:1428] Setting key ccf461d8734a45ea15cbd11b40ea9d4c
I20260927 10:47:58.073069 21375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf9c1bd605be4abe9b7f0569774fa627: Bootstrap starting.
I20260927 10:47:58.076642 21375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf9c1bd605be4abe9b7f0569774fa627: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:58.077739 21375 log.cc:824] T 00000000000000000000000000000000 P cf9c1bd605be4abe9b7f0569774fa627: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:58.080458 21375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf9c1bd605be4abe9b7f0569774fa627: No bootstrap required, opened a new log
Sep 27 10:47:58 dist-test-slave-dzhw krb5kdc[21294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 10:47:58.085565 21375 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf9c1bd605be4abe9b7f0569774fa627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf9c1bd605be4abe9b7f0569774fa627" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36489 } }
I20260927 10:47:58.085907 21375 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf9c1bd605be4abe9b7f0569774fa627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:58.085987 21375 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf9c1bd605be4abe9b7f0569774fa627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf9c1bd605be4abe9b7f0569774fa627, State: Initialized, Role: FOLLOWER
I20260927 10:47:58.086648 21375 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf9c1bd605be4abe9b7f0569774fa627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf9c1bd605be4abe9b7f0569774fa627" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36489 } }
I20260927 10:47:58.086840 21375 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf9c1bd605be4abe9b7f0569774fa627 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:47:58.086922 21375 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf9c1bd605be4abe9b7f0569774fa627 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:47:58.087041 21375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf9c1bd605be4abe9b7f0569774fa627 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:58.088507 21375 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf9c1bd605be4abe9b7f0569774fa627 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf9c1bd605be4abe9b7f0569774fa627" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36489 } }
I20260927 10:47:58.088986 21375 leader_election.cc:302] T 00000000000000000000000000000000 P cf9c1bd605be4abe9b7f0569774fa627 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf9c1bd605be4abe9b7f0569774fa627; no voters:
I20260927 10:47:58.089424 21375 leader_election.cc:288] T 00000000000000000000000000000000 P cf9c1bd605be4abe9b7f0569774fa627 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:47:58.089604 21380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf9c1bd605be4abe9b7f0569774fa627 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:58.089869 21380 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf9c1bd605be4abe9b7f0569774fa627 [term 1 LEADER]: Becoming Leader. State: Replica: cf9c1bd605be4abe9b7f0569774fa627, State: Running, Role: LEADER
I20260927 10:47:58.090145 21378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:58.073176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:38315 (local address 127.0.0.1:36489)
0927 10:47:58.073744 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:58.073768 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:58.073823 (+ 55us) server_negotiation.cc:407] Connection header received
0927 10:47:58.074912 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:58.074941 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:58.075494 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:58.075965 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:58.076981 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:58.078552 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:58.079593 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:58.080311 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:58.080510 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:58.083790 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:58.083800 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:58.083827 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:58.083870 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:58.086191 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:58.086832 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:58.086839 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:58.086847 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:58.087009 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:58.087379 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:58.087382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:58.087384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:58.088046 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:58.088301 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:58.088572 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 10:47:58.088893 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":156,"threads_started":1}
I20260927 10:47:58.090333 21380 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf9c1bd605be4abe9b7f0569774fa627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf9c1bd605be4abe9b7f0569774fa627" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36489 } }
I20260927 10:47:58.091033 21375 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf9c1bd605be4abe9b7f0569774fa627 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:47:58.092355 21382 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf9c1bd605be4abe9b7f0569774fa627 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf9c1bd605be4abe9b7f0569774fa627. Latest consensus state: current_term: 1 leader_uuid: "cf9c1bd605be4abe9b7f0569774fa627" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf9c1bd605be4abe9b7f0569774fa627" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36489 } } }
I20260927 10:47:58.092389 21381 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf9c1bd605be4abe9b7f0569774fa627 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf9c1bd605be4abe9b7f0569774fa627" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf9c1bd605be4abe9b7f0569774fa627" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36489 } } }
I20260927 10:47:58.092509 21381 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf9c1bd605be4abe9b7f0569774fa627 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:58.092928 21389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:47:58.092487 21382 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf9c1bd605be4abe9b7f0569774fa627 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:58.094667 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:47:58.096699 21389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:47:58.103590 21389 catalog_manager.cc:1347] Generated new cluster ID: 83c43dd601d84687b0c0b0b6775f5d17
I20260927 10:47:58.103720 21389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:47:58.111865 21378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:58.097512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:54881 (local address 127.0.0.1:36489)
0927 10:47:58.097653 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:58.097658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:58.101992 (+ 4334us) server_negotiation.cc:407] Connection header received
0927 10:47:58.102172 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:58.102175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:58.102237 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:58.102332 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:58.103362 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:58.104238 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:58.105494 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:58.105851 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:58.105954 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:58.108014 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:58.108019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:58.108022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:58.108048 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:58.109870 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:58.110444 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:58.110448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:58.110450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:58.110536 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:58.110912 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:58.110915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:58.110916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:58.111145 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:58.111292 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:58.111577 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 10:47:58.111683 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:47:58.120502 21389 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:47:58.121719 21389 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:47:58.133952 21389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf9c1bd605be4abe9b7f0569774fa627: Generated new TSK 0
I20260927 10:47:58.135066 21389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:47:58.145087 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 21307
W20260927 10:47:58.152822 21292 connection.cc:535] client connection to 127.0.0.1:36489 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36489): Connection reset by peer (error 104)
2026-09-27T10:47:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1557 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:47:58 dist-test-slave-dzhw krb5kdc[21406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:47:58 dist-test-slave-dzhw krb5kdc[21406](info): set up 2 sockets
Sep 27 10:47:58 dist-test-slave-dzhw krb5kdc[21406](info): commencing operation
krb5kdc: starting...
W20260927 10:47:59.259734 392 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:47:59 dist-test-slave-dzhw krb5kdc[21406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:47:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:47:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:47:59.402733 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33987
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:59.539202 21419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:59.539599 21419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:59.539650 21419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:59.544279 21419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:59.544376 21419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:59.544401 21419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:59.544426 21419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:59.544445 21419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:59.550690 21419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:47:59.552173 21419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:59.553500 21419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:59.564244 21427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:59.564303 21419 server_base.cc:1034] running on GCE node
W20260927 10:47:59.564244 21424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:59.564246 21425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:59.565335 21419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:59.566804 21419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:59.567999 21419 hybrid_clock.cc:648] HybridClock initialized: now 1790506079567969 us; error 47 us; skew 500 ppm
Sep 27 10:47:59 dist-test-slave-dzhw krb5kdc[21406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:59.576006 21419 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:47:59.577339 21419 webserver.cc:460] Webserver started at http://127.0.0.1:41949/ using document root <none> and password file <none>
I20260927 10:47:59.578152 21419 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:59.578271 21419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:59.578636 21419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:59.581439 21419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "824696125d1d41ab94d6d1ce32c617cb"
format_stamp: "Formatted at 2026-09-27 10:47:59 on dist-test-slave-dzhw"
server_key: "974668638da4462d5fd3907a292ac21b"
server_key_iv: "b1b14b7521c1f8ca289d54fca2f7a3d0"
server_key_version: "encryptionkey@0"
I20260927 10:47:59.582161 21419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "824696125d1d41ab94d6d1ce32c617cb"
format_stamp: "Formatted at 2026-09-27 10:47:59 on dist-test-slave-dzhw"
server_key: "974668638da4462d5fd3907a292ac21b"
server_key_iv: "b1b14b7521c1f8ca289d54fca2f7a3d0"
server_key_version: "encryptionkey@0"
I20260927 10:47:59.587196 21419 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 10:47:59.590349 21434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:59.592048 21419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:47:59.592235 21419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "824696125d1d41ab94d6d1ce32c617cb"
format_stamp: "Formatted at 2026-09-27 10:47:59 on dist-test-slave-dzhw"
server_key: "974668638da4462d5fd3907a292ac21b"
server_key_iv: "b1b14b7521c1f8ca289d54fca2f7a3d0"
server_key_version: "encryptionkey@0"
I20260927 10:47:59.592391 21419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:59.644186 21419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:59.648054 21419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:59.648324 21419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:59.657398 21419 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33987
I20260927 10:47:59.657418 21486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33987 every 8 connection(s)
I20260927 10:47:59.658730 21419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:47:59.662286 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 21419
I20260927 10:47:59.662477 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:47:59.662536 21487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:59.662878 392 external_mini_cluster.cc:1428] Setting key bd6c4249a78e6c0775f9ba500300e831
I20260927 10:47:59.670635 21487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 824696125d1d41ab94d6d1ce32c617cb: Bootstrap starting.
I20260927 10:47:59.673733 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:59.664643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:48473 (local address 127.0.0.1:33987)
0927 10:47:59.665209 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:59.665227 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:59.665273 (+ 46us) server_negotiation.cc:407] Connection header received
0927 10:47:59.666043 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:59.666066 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:59.666623 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:59.667064 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:59.667576 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:59.668751 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:59.669720 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:59.670259 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:59.670502 (+ 243us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:59.671614 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:59.671620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:59.671638 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:47:59.672140 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:59.672210 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:47:59.672239 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:59.672396 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:59.672584 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 10:47:59.672770 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":178,"threads_started":1}
I20260927 10:47:59.673745 21487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 824696125d1d41ab94d6d1ce32c617cb: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:59.674821 21487 log.cc:824] T 00000000000000000000000000000000 P 824696125d1d41ab94d6d1ce32c617cb: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:59.677683 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:47:59.677708 21487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 824696125d1d41ab94d6d1ce32c617cb: No bootstrap required, opened a new log
I20260927 10:47:59.683095 21487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 824696125d1d41ab94d6d1ce32c617cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824696125d1d41ab94d6d1ce32c617cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33987 } }
I20260927 10:47:59.683496 21487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 824696125d1d41ab94d6d1ce32c617cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:59.683571 21487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 824696125d1d41ab94d6d1ce32c617cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 824696125d1d41ab94d6d1ce32c617cb, State: Initialized, Role: FOLLOWER
I20260927 10:47:59.684206 21487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 824696125d1d41ab94d6d1ce32c617cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824696125d1d41ab94d6d1ce32c617cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33987 } }
I20260927 10:47:59.684404 21487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 824696125d1d41ab94d6d1ce32c617cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:47:59.684491 21487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 824696125d1d41ab94d6d1ce32c617cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:47:59.684646 21487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 824696125d1d41ab94d6d1ce32c617cb [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:59.685444 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:59.680344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:50323 (local address 127.0.0.1:33987)
0927 10:47:59.680520 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:59.680527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:59.680840 (+ 313us) server_negotiation.cc:407] Connection header received
0927 10:47:59.680982 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:59.680986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:59.681051 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:59.681149 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:59.681464 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:59.682307 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:59.683372 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:59.683731 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:59.683850 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:59.684951 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:59.684958 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:59.684965 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:47:59.684984 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:59.685020 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:47:59.685035 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:59.685126 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:59.685244 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 10:47:59.685286 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:47:59.686285 21487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 824696125d1d41ab94d6d1ce32c617cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824696125d1d41ab94d6d1ce32c617cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33987 } }
I20260927 10:47:59.686805 21487 leader_election.cc:302] T 00000000000000000000000000000000 P 824696125d1d41ab94d6d1ce32c617cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 824696125d1d41ab94d6d1ce32c617cb; no voters:
I20260927 10:47:59.687282 21487 leader_election.cc:288] T 00000000000000000000000000000000 P 824696125d1d41ab94d6d1ce32c617cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:47:59.687422 21497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 824696125d1d41ab94d6d1ce32c617cb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:59.687726 21497 raft_consensus.cc:686] T 00000000000000000000000000000000 P 824696125d1d41ab94d6d1ce32c617cb [term 1 LEADER]: Becoming Leader. State: Replica: 824696125d1d41ab94d6d1ce32c617cb, State: Running, Role: LEADER
I20260927 10:47:59.688140 21497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 824696125d1d41ab94d6d1ce32c617cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824696125d1d41ab94d6d1ce32c617cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33987 } }
I20260927 10:47:59.688908 21487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 824696125d1d41ab94d6d1ce32c617cb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:47:59.690349 21498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824696125d1d41ab94d6d1ce32c617cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "824696125d1d41ab94d6d1ce32c617cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824696125d1d41ab94d6d1ce32c617cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33987 } } }
I20260927 10:47:59.690529 21498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824696125d1d41ab94d6d1ce32c617cb [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:59.690584 21499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824696125d1d41ab94d6d1ce32c617cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 824696125d1d41ab94d6d1ce32c617cb. Latest consensus state: current_term: 1 leader_uuid: "824696125d1d41ab94d6d1ce32c617cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824696125d1d41ab94d6d1ce32c617cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33987 } } }
I20260927 10:47:59.690680 21499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824696125d1d41ab94d6d1ce32c617cb [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:59.691982 21506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:47:59.696422 21506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:47:59.703940 21506 catalog_manager.cc:1347] Generated new cluster ID: 398beb14211f4f8faa1c31b7881f5565
I20260927 10:47:59.704056 21506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:47:59.712373 21506 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:47:59.713588 21506 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:47:59.722365 21506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 824696125d1d41ab94d6d1ce32c617cb: Generated new TSK 0
I20260927 10:47:59.723399 21506 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:47:59.730598 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 21419
W20260927 10:47:59.737461 21404 connection.cc:535] client connection to 127.0.0.1:33987 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33987): Connection reset by peer (error 104)
2026-09-27T10:47:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1584 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:47:59 dist-test-slave-dzhw krb5kdc[21518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:47:59 dist-test-slave-dzhw krb5kdc[21518](info): set up 2 sockets
Sep 27 10:47:59 dist-test-slave-dzhw krb5kdc[21518](info): commencing operation
krb5kdc: starting...
W20260927 10:48:00.803454 392 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:48:00 dist-test-slave-dzhw krb5kdc[21518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:48:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:48:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:48:00.937125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32881
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:01.068668 21531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:01.069068 21531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:01.069157 21531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:01.073956 21531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:01.074095 21531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:01.074138 21531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:01.074182 21531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:01.074218 21531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:01.080932 21531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:48:01.082470 21531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:01.083827 21531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:01.094739 21536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:01.094933 21539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:01.094738 21537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:01.095093 21531 server_base.cc:1034] running on GCE node
I20260927 10:48:01.095994 21531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:01.097401 21531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:01.098651 21531 hybrid_clock.cc:648] HybridClock initialized: now 1790506081098617 us; error 59 us; skew 500 ppm
Sep 27 10:48:01 dist-test-slave-dzhw krb5kdc[21518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:01.101810 21531 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:48:01.102882 21531 webserver.cc:460] Webserver started at http://127.0.0.1:35139/ using document root <none> and password file <none>
I20260927 10:48:01.103516 21531 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:01.103581 21531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:01.103780 21531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:01.106102 21531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "8a19a1cc4ad94af78cc820d7954498e5"
format_stamp: "Formatted at 2026-09-27 10:48:01 on dist-test-slave-dzhw"
server_key: "22fbbc3394e5cdefafa5a8322a9025f1"
server_key_iv: "ba22e6deb713cab92eab63f3fa182639"
server_key_version: "encryptionkey@0"
I20260927 10:48:01.106736 21531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "8a19a1cc4ad94af78cc820d7954498e5"
format_stamp: "Formatted at 2026-09-27 10:48:01 on dist-test-slave-dzhw"
server_key: "22fbbc3394e5cdefafa5a8322a9025f1"
server_key_iv: "ba22e6deb713cab92eab63f3fa182639"
server_key_version: "encryptionkey@0"
I20260927 10:48:01.111240 21531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 10:48:01.114105 21546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:01.115808 21531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:48:01.115978 21531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "8a19a1cc4ad94af78cc820d7954498e5"
format_stamp: "Formatted at 2026-09-27 10:48:01 on dist-test-slave-dzhw"
server_key: "22fbbc3394e5cdefafa5a8322a9025f1"
server_key_iv: "ba22e6deb713cab92eab63f3fa182639"
server_key_version: "encryptionkey@0"
I20260927 10:48:01.116137 21531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:01.121392 21531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:01.126123 21531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:01.126341 21531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:01.136140 21531 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32881
I20260927 10:48:01.136199 21598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32881 every 8 connection(s)
I20260927 10:48:01.137246 21531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:48:01.141081 21599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:01.145242 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 21531
I20260927 10:48:01.145373 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:48:01.145730 392 external_mini_cluster.cc:1428] Setting key 08d19619becfe7c5858f821800ba0fdb
I20260927 10:48:01.148887 21599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a19a1cc4ad94af78cc820d7954498e5: Bootstrap starting.
I20260927 10:48:01.152164 21599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a19a1cc4ad94af78cc820d7954498e5: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:01.152354 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:01.147397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:37283 (local address 127.0.0.1:32881)
0927 10:48:01.148054 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:01.148075 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:01.148124 (+ 49us) server_negotiation.cc:407] Connection header received
0927 10:48:01.149014 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:01.149067 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:01.149487 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:01.149918 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:01.150870 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:01.150876 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:01.150890 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:01.150901 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:01.150960 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:01.150982 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:01.151158 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:01.151248 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 10:48:01.151430 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":203,"threads_started":1}
I20260927 10:48:01.153326 21599 log.cc:824] T 00000000000000000000000000000000 P 8a19a1cc4ad94af78cc820d7954498e5: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:01.155934 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:48:01.156129 21599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a19a1cc4ad94af78cc820d7954498e5: No bootstrap required, opened a new log
I20260927 10:48:01.160799 21599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a19a1cc4ad94af78cc820d7954498e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a19a1cc4ad94af78cc820d7954498e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32881 } }
I20260927 10:48:01.161171 21599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a19a1cc4ad94af78cc820d7954498e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:01.161262 21599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a19a1cc4ad94af78cc820d7954498e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a19a1cc4ad94af78cc820d7954498e5, State: Initialized, Role: FOLLOWER
I20260927 10:48:01.161931 21599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a19a1cc4ad94af78cc820d7954498e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a19a1cc4ad94af78cc820d7954498e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32881 } }
I20260927 10:48:01.162155 21599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a19a1cc4ad94af78cc820d7954498e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:48:01.162250 21599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a19a1cc4ad94af78cc820d7954498e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:48:01.162406 21599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a19a1cc4ad94af78cc820d7954498e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:01.163341 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:01.158463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:37417 (local address 127.0.0.1:32881)
0927 10:48:01.158570 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:01.158573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:01.161599 (+ 3026us) server_negotiation.cc:407] Connection header received
0927 10:48:01.161792 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:01.161794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:01.161838 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:01.161911 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:01.162875 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:01.162878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:01.162880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:01.162891 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:01.162919 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:01.162930 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:01.163027 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:01.163138 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 10:48:01.163176 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 10:48:01.164083 21599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a19a1cc4ad94af78cc820d7954498e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a19a1cc4ad94af78cc820d7954498e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32881 } }
I20260927 10:48:01.164587 21599 leader_election.cc:302] T 00000000000000000000000000000000 P 8a19a1cc4ad94af78cc820d7954498e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a19a1cc4ad94af78cc820d7954498e5; no voters:
I20260927 10:48:01.165001 21599 leader_election.cc:288] T 00000000000000000000000000000000 P 8a19a1cc4ad94af78cc820d7954498e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:48:01.165182 21609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a19a1cc4ad94af78cc820d7954498e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:01.165491 21609 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a19a1cc4ad94af78cc820d7954498e5 [term 1 LEADER]: Becoming Leader. State: Replica: 8a19a1cc4ad94af78cc820d7954498e5, State: Running, Role: LEADER
I20260927 10:48:01.165977 21609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a19a1cc4ad94af78cc820d7954498e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a19a1cc4ad94af78cc820d7954498e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32881 } }
I20260927 10:48:01.166741 21599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a19a1cc4ad94af78cc820d7954498e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:48:01.168331 21610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a19a1cc4ad94af78cc820d7954498e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a19a1cc4ad94af78cc820d7954498e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a19a1cc4ad94af78cc820d7954498e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32881 } } }
I20260927 10:48:01.168383 21611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a19a1cc4ad94af78cc820d7954498e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a19a1cc4ad94af78cc820d7954498e5. Latest consensus state: current_term: 1 leader_uuid: "8a19a1cc4ad94af78cc820d7954498e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a19a1cc4ad94af78cc820d7954498e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32881 } } }
I20260927 10:48:01.168509 21611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a19a1cc4ad94af78cc820d7954498e5 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:01.168509 21610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a19a1cc4ad94af78cc820d7954498e5 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:01.168987 21618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:48:01.173010 21618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:48:01.180768 21618 catalog_manager.cc:1347] Generated new cluster ID: 17af7cc540704c7eb678224cbc3b0acc
I20260927 10:48:01.180878 21618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:48:01.219822 21618 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:48:01.220242 21618 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:48:01.231360 21618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a19a1cc4ad94af78cc820d7954498e5: Generated new TSK 0
I20260927 10:48:01.232131 21618 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:48:01.235693 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 21531
2026-09-27T10:48:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1503 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:48:01 dist-test-slave-dzhw krb5kdc[21630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:48:01 dist-test-slave-dzhw krb5kdc[21630](info): set up 2 sockets
Sep 27 10:48:01 dist-test-slave-dzhw krb5kdc[21630](info): commencing operation
krb5kdc: starting...
W20260927 10:48:02.295667 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:48:02 dist-test-slave-dzhw krb5kdc[21630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:48:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:48:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:48:02.452407 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39347
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:02.584640 21643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:02.584985 21643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:02.585031 21643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:02.589356 21643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:02.589442 21643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:02.589462 21643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:02.589479 21643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:02.589495 21643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:02.595299 21643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39347
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:48:02.596710 21643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:02.598063 21643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:48:02.608520 21643 server_base.cc:1034] running on GCE node
W20260927 10:48:02.608511 21648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:02.608577 21649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:02.609434 21651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:02.610159 21643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:02.611937 21643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:02.613123 21643 hybrid_clock.cc:648] HybridClock initialized: now 1790506082613097 us; error 42 us; skew 500 ppm
Sep 27 10:48:02 dist-test-slave-dzhw krb5kdc[21630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:02.616238 21643 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:48:02.617301 21643 webserver.cc:460] Webserver started at http://127.0.0.1:37503/ using document root <none> and password file <none>
I20260927 10:48:02.617965 21643 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:02.618029 21643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:02.618330 21643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:02.620740 21643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "2edaf25d91fc4fd0b8ce7a7a9ab45fad"
format_stamp: "Formatted at 2026-09-27 10:48:02 on dist-test-slave-dzhw"
server_key: "06d1164dda69e0ab9242a87d7e4d9624"
server_key_iv: "8b8da1caa26b2ff4202f398765791d51"
server_key_version: "encryptionkey@0"
I20260927 10:48:02.621405 21643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "2edaf25d91fc4fd0b8ce7a7a9ab45fad"
format_stamp: "Formatted at 2026-09-27 10:48:02 on dist-test-slave-dzhw"
server_key: "06d1164dda69e0ab9242a87d7e4d9624"
server_key_iv: "8b8da1caa26b2ff4202f398765791d51"
server_key_version: "encryptionkey@0"
I20260927 10:48:02.626230 21643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 10:48:02.629022 21658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:02.630511 21643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:48:02.630652 21643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "2edaf25d91fc4fd0b8ce7a7a9ab45fad"
format_stamp: "Formatted at 2026-09-27 10:48:02 on dist-test-slave-dzhw"
server_key: "06d1164dda69e0ab9242a87d7e4d9624"
server_key_iv: "8b8da1caa26b2ff4202f398765791d51"
server_key_version: "encryptionkey@0"
I20260927 10:48:02.630781 21643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:02.651438 21643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:02.655166 21643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:02.655411 21643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:02.663805 21643 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39347
I20260927 10:48:02.663823 21710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39347 every 8 connection(s)
I20260927 10:48:02.664973 21643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:48:02.668519 21711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:02.671269 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 21643
I20260927 10:48:02.671452 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:48:02.671787 392 external_mini_cluster.cc:1428] Setting key 2cfb3c67f043ca81b86882575467bc0e
I20260927 10:48:02.676509 21711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2edaf25d91fc4fd0b8ce7a7a9ab45fad: Bootstrap starting.
I20260927 10:48:02.679879 21711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2edaf25d91fc4fd0b8ce7a7a9ab45fad: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:02.680994 21711 log.cc:824] T 00000000000000000000000000000000 P 2edaf25d91fc4fd0b8ce7a7a9ab45fad: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:48:02 dist-test-slave-dzhw krb5kdc[21630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 10:48:02.683983 21711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2edaf25d91fc4fd0b8ce7a7a9ab45fad: No bootstrap required, opened a new log
I20260927 10:48:02.689664 21711 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2edaf25d91fc4fd0b8ce7a7a9ab45fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edaf25d91fc4fd0b8ce7a7a9ab45fad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39347 } }
I20260927 10:48:02.690011 21711 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2edaf25d91fc4fd0b8ce7a7a9ab45fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:02.690061 21711 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2edaf25d91fc4fd0b8ce7a7a9ab45fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edaf25d91fc4fd0b8ce7a7a9ab45fad, State: Initialized, Role: FOLLOWER
I20260927 10:48:02.690588 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:02.673545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:54801 (local address 127.0.0.1:39347)
0927 10:48:02.674096 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:02.674121 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:02.674172 (+ 51us) server_negotiation.cc:407] Connection header received
0927 10:48:02.675157 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:02.675188 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:02.675734 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:02.676176 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:02.677075 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:02.678690 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:02.679653 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:02.680428 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:02.680593 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:02.684119 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:02.684126 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:02.684145 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:02.684181 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:02.686435 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:02.687098 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:02.687106 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:02.687113 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:02.687281 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:02.687706 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:02.687710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:02.687712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:02.688373 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:02.688616 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:02.689033 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 10:48:02.689329 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":144,"threads_started":1}
I20260927 10:48:02.690691 21711 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2edaf25d91fc4fd0b8ce7a7a9ab45fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edaf25d91fc4fd0b8ce7a7a9ab45fad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39347 } }
I20260927 10:48:02.690862 21711 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2edaf25d91fc4fd0b8ce7a7a9ab45fad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:48:02.690939 21711 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2edaf25d91fc4fd0b8ce7a7a9ab45fad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:48:02.691051 21711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2edaf25d91fc4fd0b8ce7a7a9ab45fad [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:02.692476 21711 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2edaf25d91fc4fd0b8ce7a7a9ab45fad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edaf25d91fc4fd0b8ce7a7a9ab45fad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39347 } }
I20260927 10:48:02.692948 21711 leader_election.cc:302] T 00000000000000000000000000000000 P 2edaf25d91fc4fd0b8ce7a7a9ab45fad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2edaf25d91fc4fd0b8ce7a7a9ab45fad; no voters:
I20260927 10:48:02.693353 21711 leader_election.cc:288] T 00000000000000000000000000000000 P 2edaf25d91fc4fd0b8ce7a7a9ab45fad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:48:02.693542 21716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2edaf25d91fc4fd0b8ce7a7a9ab45fad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:02.693804 21716 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2edaf25d91fc4fd0b8ce7a7a9ab45fad [term 1 LEADER]: Becoming Leader. State: Replica: 2edaf25d91fc4fd0b8ce7a7a9ab45fad, State: Running, Role: LEADER
I20260927 10:48:02.694171 21716 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2edaf25d91fc4fd0b8ce7a7a9ab45fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edaf25d91fc4fd0b8ce7a7a9ab45fad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39347 } }
I20260927 10:48:02.694525 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:48:02.695047 21711 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2edaf25d91fc4fd0b8ce7a7a9ab45fad [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:48:02.696452 21718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2edaf25d91fc4fd0b8ce7a7a9ab45fad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2edaf25d91fc4fd0b8ce7a7a9ab45fad. Latest consensus state: current_term: 1 leader_uuid: "2edaf25d91fc4fd0b8ce7a7a9ab45fad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edaf25d91fc4fd0b8ce7a7a9ab45fad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39347 } } }
I20260927 10:48:02.696452 21717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2edaf25d91fc4fd0b8ce7a7a9ab45fad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2edaf25d91fc4fd0b8ce7a7a9ab45fad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edaf25d91fc4fd0b8ce7a7a9ab45fad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39347 } } }
I20260927 10:48:02.696630 21718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2edaf25d91fc4fd0b8ce7a7a9ab45fad [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:02.696632 21717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2edaf25d91fc4fd0b8ce7a7a9ab45fad [sys.catalog]: This master's current role is: LEADER
W20260927 10:48:02.701320 21736 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2edaf25d91fc4fd0b8ce7a7a9ab45fad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:48:02.701423 21736 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:48:02.701556 21729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:48:02.702755 21729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:48:02.707512 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:02.697257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:33227 (local address 127.0.0.1:39347)
0927 10:48:02.697411 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:02.697417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:02.697643 (+ 226us) server_negotiation.cc:407] Connection header received
0927 10:48:02.697860 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:02.697864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:02.697932 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:02.697991 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:02.699135 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:02.699940 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:02.701096 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:02.701443 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:02.701546 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:02.703438 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:02.703442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:02.703446 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:02.703479 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:02.704990 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:02.705525 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:02.705535 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:02.705537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:02.705616 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:02.706032 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:02.706046 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:02.706048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:02.706290 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:02.706459 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:02.707230 (+ 771us) server_negotiation.cc:299] Negotiation successful
0927 10:48:02.707338 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:48:02.710947 21729 catalog_manager.cc:1347] Generated new cluster ID: 56b3a9eae79e41af9e86c22b816b8808
I20260927 10:48:02.711035 21729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:48:02.728760 21729 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:48:02.729972 21729 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:48:02.741186 21729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2edaf25d91fc4fd0b8ce7a7a9ab45fad: Generated new TSK 0
I20260927 10:48:02.742339 21729 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:48:02.752389 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 21643
W20260927 10:48:02.759124 21628 connection.cc:535] client connection to 127.0.0.1:39347 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39347): Connection reset by peer (error 104)
2026-09-27T10:48:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1517 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:48:02 dist-test-slave-dzhw krb5kdc[21742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:48:02 dist-test-slave-dzhw krb5kdc[21742](info): set up 2 sockets
Sep 27 10:48:02 dist-test-slave-dzhw krb5kdc[21742](info): commencing operation
krb5kdc: starting...
W20260927 10:48:03.799305 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:48:03 dist-test-slave-dzhw krb5kdc[21742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:48:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:48:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:48:03.937644 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39023
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:04.072386 21755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:04.072809 21755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:04.072889 21755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:04.077456 21755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:04.077590 21755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:04.077630 21755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:04.077669 21755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:04.077713 21755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:04.084136 21755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39023
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:48:04.085763 21755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:04.087158 21755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:04.098249 21763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:04.098202 21761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:04.098217 21760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:04.098542 21755 server_base.cc:1034] running on GCE node
I20260927 10:48:04.099480 21755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:04.100955 21755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:04.102162 21755 hybrid_clock.cc:648] HybridClock initialized: now 1790506084102149 us; error 39 us; skew 500 ppm
Sep 27 10:48:04 dist-test-slave-dzhw krb5kdc[21742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:04.105685 21755 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:48:04.106956 21755 webserver.cc:460] Webserver started at http://127.0.0.1:43453/ using document root <none> and password file <none>
I20260927 10:48:04.107714 21755 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:04.107820 21755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:04.108110 21755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:04.110756 21755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "cde6e76db3f949bfa5e31b6a00586bdb"
format_stamp: "Formatted at 2026-09-27 10:48:04 on dist-test-slave-dzhw"
server_key: "ce4c00b94668cce85b49b75c78f062ad"
server_key_iv: "7da9c387186a6ce1cc5eb0ea533267bb"
server_key_version: "encryptionkey@0"
I20260927 10:48:04.111428 21755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "cde6e76db3f949bfa5e31b6a00586bdb"
format_stamp: "Formatted at 2026-09-27 10:48:04 on dist-test-slave-dzhw"
server_key: "ce4c00b94668cce85b49b75c78f062ad"
server_key_iv: "7da9c387186a6ce1cc5eb0ea533267bb"
server_key_version: "encryptionkey@0"
I20260927 10:48:04.116289 21755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 10:48:04.119633 21770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:04.121322 21755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:48:04.121515 21755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "cde6e76db3f949bfa5e31b6a00586bdb"
format_stamp: "Formatted at 2026-09-27 10:48:04 on dist-test-slave-dzhw"
server_key: "ce4c00b94668cce85b49b75c78f062ad"
server_key_iv: "7da9c387186a6ce1cc5eb0ea533267bb"
server_key_version: "encryptionkey@0"
I20260927 10:48:04.121690 21755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:04.134812 21755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:04.138339 21755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:04.138612 21755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:04.147931 21755 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39023
I20260927 10:48:04.147974 21822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39023 every 8 connection(s)
I20260927 10:48:04.149135 21755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:48:04.152925 21823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:04.156220 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 21755
I20260927 10:48:04.156391 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:48:04.156718 392 external_mini_cluster.cc:1428] Setting key e4662a936c42e6c271639d7652da4887
I20260927 10:48:04.160396 21823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde6e76db3f949bfa5e31b6a00586bdb: Bootstrap starting.
I20260927 10:48:04.163575 21823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cde6e76db3f949bfa5e31b6a00586bdb: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:04.164601 21823 log.cc:824] T 00000000000000000000000000000000 P cde6e76db3f949bfa5e31b6a00586bdb: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:04.167958 21823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde6e76db3f949bfa5e31b6a00586bdb: No bootstrap required, opened a new log
I20260927 10:48:04.169034 21826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:04.158742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:50945 (local address 127.0.0.1:39023)
0927 10:48:04.159324 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:04.159345 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:04.159393 (+ 48us) server_negotiation.cc:407] Connection header received
0927 10:48:04.160246 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:04.160270 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:04.160821 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:04.161227 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:04.161675 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:04.162928 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:04.164290 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:04.164981 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:04.165155 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:04.166345 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:04.166353 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:04.166375 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:04.167130 (+ 755us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:04.167241 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:04.167276 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:04.167448 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:04.167663 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 10:48:04.167844 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":196,"threads_started":1}
I20260927 10:48:04.173105 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:48:04.173120 21823 raft_consensus.cc:348] T 00000000000000000000000000000000 P cde6e76db3f949bfa5e31b6a00586bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde6e76db3f949bfa5e31b6a00586bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39023 } }
I20260927 10:48:04.173455 21823 raft_consensus.cc:374] T 00000000000000000000000000000000 P cde6e76db3f949bfa5e31b6a00586bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:04.173549 21823 raft_consensus.cc:729] T 00000000000000000000000000000000 P cde6e76db3f949bfa5e31b6a00586bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde6e76db3f949bfa5e31b6a00586bdb, State: Initialized, Role: FOLLOWER
I20260927 10:48:04.174125 21823 consensus_queue.cc:260] T 00000000000000000000000000000000 P cde6e76db3f949bfa5e31b6a00586bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde6e76db3f949bfa5e31b6a00586bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39023 } }
I20260927 10:48:04.174283 21823 raft_consensus.cc:388] T 00000000000000000000000000000000 P cde6e76db3f949bfa5e31b6a00586bdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:48:04.174373 21823 raft_consensus.cc:482] T 00000000000000000000000000000000 P cde6e76db3f949bfa5e31b6a00586bdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:48:04.174515 21823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cde6e76db3f949bfa5e31b6a00586bdb [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:04.175786 21823 raft_consensus.cc:504] T 00000000000000000000000000000000 P cde6e76db3f949bfa5e31b6a00586bdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde6e76db3f949bfa5e31b6a00586bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39023 } }
I20260927 10:48:04.176283 21823 leader_election.cc:302] T 00000000000000000000000000000000 P cde6e76db3f949bfa5e31b6a00586bdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cde6e76db3f949bfa5e31b6a00586bdb; no voters:
I20260927 10:48:04.176703 21823 leader_election.cc:288] T 00000000000000000000000000000000 P cde6e76db3f949bfa5e31b6a00586bdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:48:04.176882 21833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cde6e76db3f949bfa5e31b6a00586bdb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:04.177223 21833 raft_consensus.cc:686] T 00000000000000000000000000000000 P cde6e76db3f949bfa5e31b6a00586bdb [term 1 LEADER]: Becoming Leader. State: Replica: cde6e76db3f949bfa5e31b6a00586bdb, State: Running, Role: LEADER
I20260927 10:48:04.177707 21833 consensus_queue.cc:237] T 00000000000000000000000000000000 P cde6e76db3f949bfa5e31b6a00586bdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde6e76db3f949bfa5e31b6a00586bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39023 } }
I20260927 10:48:04.178344 21823 sys_catalog.cc:564] T 00000000000000000000000000000000 P cde6e76db3f949bfa5e31b6a00586bdb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:48:04.179502 21833 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde6e76db3f949bfa5e31b6a00586bdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader cde6e76db3f949bfa5e31b6a00586bdb. Latest consensus state: current_term: 1 leader_uuid: "cde6e76db3f949bfa5e31b6a00586bdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde6e76db3f949bfa5e31b6a00586bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39023 } } }
I20260927 10:48:04.179688 21833 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde6e76db3f949bfa5e31b6a00586bdb [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:04.180154 21842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:48:04.183264 21834 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde6e76db3f949bfa5e31b6a00586bdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cde6e76db3f949bfa5e31b6a00586bdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde6e76db3f949bfa5e31b6a00586bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39023 } } }
I20260927 10:48:04.183495 21834 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde6e76db3f949bfa5e31b6a00586bdb [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:04.186142 21826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:04.175908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:41851 (local address 127.0.0.1:39023)
0927 10:48:04.176092 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:04.176097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:04.176199 (+ 102us) server_negotiation.cc:407] Connection header received
0927 10:48:04.176325 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:04.176328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:04.176398 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:04.176512 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:04.176920 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:04.177736 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:04.183128 (+ 5392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:04.183504 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:04.183620 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:04.185075 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:04.185083 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:04.185089 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:04.185110 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:04.185189 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:04.185204 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:04.185368 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:04.185773 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 10:48:04.185856 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 10:48:04.186546 21842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:48:04.194355 21842 catalog_manager.cc:1347] Generated new cluster ID: 1432fdcf56554030994106cdcc42acf2
I20260927 10:48:04.194475 21842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:48:04.203917 21842 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:48:04.205075 21842 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:48:04.210148 21842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cde6e76db3f949bfa5e31b6a00586bdb: Generated new TSK 0
I20260927 10:48:04.211122 21842 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:48:04.217973 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 21755
W20260927 10:48:04.224061 21740 connection.cc:535] client connection to 127.0.0.1:39023 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39023): Connection reset by peer (error 104)
2026-09-27T10:48:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1464 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:48:04 dist-test-slave-dzhw krb5kdc[21854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:48:04 dist-test-slave-dzhw krb5kdc[21854](info): set up 2 sockets
Sep 27 10:48:04 dist-test-slave-dzhw krb5kdc[21854](info): commencing operation
krb5kdc: starting...
W20260927 10:48:05.259157 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:48:05 dist-test-slave-dzhw krb5kdc[21854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:48:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:48:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:48:05.422303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34907
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:05.557112 21867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:05.557571 21867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:05.557663 21867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:05.562109 21867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:05.562225 21867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:05.562265 21867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:05.562325 21867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:05.562368 21867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:05.568332 21867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34907
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:48:05.569813 21867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:05.571163 21867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:05.581630 21873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:05.581619 21872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:05.581697 21867 server_base.cc:1034] running on GCE node
W20260927 10:48:05.581617 21875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:05.582684 21867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:05.584182 21867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:05.585391 21867 hybrid_clock.cc:648] HybridClock initialized: now 1790506085585363 us; error 43 us; skew 500 ppm
Sep 27 10:48:05 dist-test-slave-dzhw krb5kdc[21854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:05.596825 21867 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:48:05.598223 21867 webserver.cc:460] Webserver started at http://127.0.0.1:33231/ using document root <none> and password file <none>
I20260927 10:48:05.599026 21867 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:05.599152 21867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:05.599458 21867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:05.602222 21867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "36675980c4b248cca1b0218f8901f0e4"
format_stamp: "Formatted at 2026-09-27 10:48:05 on dist-test-slave-dzhw"
server_key: "d5d657eec799a7c33f4c6c646e900724"
server_key_iv: "c3265ecdc47cf0db9f45c2b2b023d9e4"
server_key_version: "encryptionkey@0"
I20260927 10:48:05.602922 21867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "36675980c4b248cca1b0218f8901f0e4"
format_stamp: "Formatted at 2026-09-27 10:48:05 on dist-test-slave-dzhw"
server_key: "d5d657eec799a7c33f4c6c646e900724"
server_key_iv: "c3265ecdc47cf0db9f45c2b2b023d9e4"
server_key_version: "encryptionkey@0"
I20260927 10:48:05.607954 21867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 10:48:05.611609 21882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:05.613282 21867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:48:05.613473 21867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "36675980c4b248cca1b0218f8901f0e4"
format_stamp: "Formatted at 2026-09-27 10:48:05 on dist-test-slave-dzhw"
server_key: "d5d657eec799a7c33f4c6c646e900724"
server_key_iv: "c3265ecdc47cf0db9f45c2b2b023d9e4"
server_key_version: "encryptionkey@0"
I20260927 10:48:05.613634 21867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:05.619247 21867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:05.623168 21867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:05.623409 21867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:05.631765 21867 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34907
I20260927 10:48:05.631785 21934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34907 every 8 connection(s)
I20260927 10:48:05.632984 21867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:48:05.636634 21935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:05.640986 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 21867
I20260927 10:48:05.641155 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:48:05.641501 392 external_mini_cluster.cc:1428] Setting key fffc7dc4edb38de91566464e44ba2d0e
I20260927 10:48:05.644626 21935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36675980c4b248cca1b0218f8901f0e4: Bootstrap starting.
I20260927 10:48:05.647675 21935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36675980c4b248cca1b0218f8901f0e4: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:05.647872 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:05.643333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:55685 (local address 127.0.0.1:34907)
0927 10:48:05.643877 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:05.643895 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:05.643938 (+ 43us) server_negotiation.cc:407] Connection header received
0927 10:48:05.644674 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:05.644725 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:05.645107 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:05.645559 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:05.646465 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:05.646472 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:05.646495 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:05.646507 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:05.646567 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:05.646590 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:05.646744 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:05.646820 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 10:48:05.647012 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":172,"threads_started":1}
I20260927 10:48:05.648597 21935 log.cc:824] T 00000000000000000000000000000000 P 36675980c4b248cca1b0218f8901f0e4: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:05.651150 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:48:05.651166 21935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36675980c4b248cca1b0218f8901f0e4: No bootstrap required, opened a new log
I20260927 10:48:05.655728 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:05.653497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:55841 (local address 127.0.0.1:34907)
0927 10:48:05.653639 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:05.653644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:05.653867 (+ 223us) server_negotiation.cc:407] Connection header received
0927 10:48:05.654009 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:05.654012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:05.654048 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:05.654153 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:05.655286 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:05.655290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:05.655294 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:05.655306 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:05.655342 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:05.655357 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:05.655454 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:05.655547 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 10:48:05.655594 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:48:05.656231 21935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36675980c4b248cca1b0218f8901f0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36675980c4b248cca1b0218f8901f0e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34907 } }
I20260927 10:48:05.656564 21935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36675980c4b248cca1b0218f8901f0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:05.656633 21935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36675980c4b248cca1b0218f8901f0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36675980c4b248cca1b0218f8901f0e4, State: Initialized, Role: FOLLOWER
I20260927 10:48:05.657197 21935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36675980c4b248cca1b0218f8901f0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36675980c4b248cca1b0218f8901f0e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34907 } }
I20260927 10:48:05.657375 21935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36675980c4b248cca1b0218f8901f0e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:48:05.657478 21935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36675980c4b248cca1b0218f8901f0e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:48:05.657591 21935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36675980c4b248cca1b0218f8901f0e4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:05.658859 21935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36675980c4b248cca1b0218f8901f0e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36675980c4b248cca1b0218f8901f0e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34907 } }
I20260927 10:48:05.659308 21935 leader_election.cc:302] T 00000000000000000000000000000000 P 36675980c4b248cca1b0218f8901f0e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36675980c4b248cca1b0218f8901f0e4; no voters:
I20260927 10:48:05.659727 21935 leader_election.cc:288] T 00000000000000000000000000000000 P 36675980c4b248cca1b0218f8901f0e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:48:05.659919 21945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36675980c4b248cca1b0218f8901f0e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:05.660167 21945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36675980c4b248cca1b0218f8901f0e4 [term 1 LEADER]: Becoming Leader. State: Replica: 36675980c4b248cca1b0218f8901f0e4, State: Running, Role: LEADER
I20260927 10:48:05.660645 21945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36675980c4b248cca1b0218f8901f0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36675980c4b248cca1b0218f8901f0e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34907 } }
I20260927 10:48:05.661432 21935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36675980c4b248cca1b0218f8901f0e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:48:05.662806 21947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36675980c4b248cca1b0218f8901f0e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36675980c4b248cca1b0218f8901f0e4. Latest consensus state: current_term: 1 leader_uuid: "36675980c4b248cca1b0218f8901f0e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36675980c4b248cca1b0218f8901f0e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34907 } } }
I20260927 10:48:05.662948 21947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36675980c4b248cca1b0218f8901f0e4 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:05.663395 21954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:48:05.663354 21946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36675980c4b248cca1b0218f8901f0e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36675980c4b248cca1b0218f8901f0e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36675980c4b248cca1b0218f8901f0e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34907 } } }
I20260927 10:48:05.663491 21946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36675980c4b248cca1b0218f8901f0e4 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:05.667312 21954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:48:05.674266 21954 catalog_manager.cc:1347] Generated new cluster ID: 157ec7bee7624007b30cccc32a9e6004
I20260927 10:48:05.674371 21954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:48:05.692975 21954 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:48:05.693454 21954 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:48:05.701428 21954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36675980c4b248cca1b0218f8901f0e4: Generated new TSK 0
I20260927 10:48:05.702601 21954 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:48:05.714001 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 21867
2026-09-27T10:48:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1497 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:48:05 dist-test-slave-dzhw krb5kdc[21966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:48:05 dist-test-slave-dzhw krb5kdc[21966](info): set up 2 sockets
Sep 27 10:48:05 dist-test-slave-dzhw krb5kdc[21966](info): commencing operation
krb5kdc: starting...
W20260927 10:48:06.775272 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:48:06 dist-test-slave-dzhw krb5kdc[21966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:48:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:48:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:48:06.918880 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44949
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:07.055981 21979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:07.056453 21979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:07.056627 21979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:07.061779 21979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:07.062153 21979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:07.062295 21979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:07.062353 21979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:07.062405 21979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:07.070946 21979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44949
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:48:07.072757 21979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:07.074754 21979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:07.086961 21985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:07.086952 21987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:07.086952 21984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:07.087272 21979 server_base.cc:1034] running on GCE node
I20260927 10:48:07.088160 21979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:07.089769 21979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:07.090935 21979 hybrid_clock.cc:648] HybridClock initialized: now 1790506087090917 us; error 38 us; skew 500 ppm
Sep 27 10:48:07 dist-test-slave-dzhw krb5kdc[21966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:07.095010 21979 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:48:07.096509 21979 webserver.cc:460] Webserver started at http://127.0.98.62:42093/ using document root <none> and password file <none>
I20260927 10:48:07.097378 21979 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:07.097491 21979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:07.097802 21979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:07.100737 21979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "cc46e1a1e7a3499e9eb4f5e02b0bb53a"
format_stamp: "Formatted at 2026-09-27 10:48:07 on dist-test-slave-dzhw"
server_key: "a646b6c4a6bbe47350d411fd69a3754a"
server_key_iv: "e5954d125246ccae5666afecca30f3dd"
server_key_version: "encryptionkey@0"
I20260927 10:48:07.101534 21979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "cc46e1a1e7a3499e9eb4f5e02b0bb53a"
format_stamp: "Formatted at 2026-09-27 10:48:07 on dist-test-slave-dzhw"
server_key: "a646b6c4a6bbe47350d411fd69a3754a"
server_key_iv: "e5954d125246ccae5666afecca30f3dd"
server_key_version: "encryptionkey@0"
I20260927 10:48:07.106933 21979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 10:48:07.110491 21994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:07.112443 21979 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:48:07.112623 21979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "cc46e1a1e7a3499e9eb4f5e02b0bb53a"
format_stamp: "Formatted at 2026-09-27 10:48:07 on dist-test-slave-dzhw"
server_key: "a646b6c4a6bbe47350d411fd69a3754a"
server_key_iv: "e5954d125246ccae5666afecca30f3dd"
server_key_version: "encryptionkey@0"
I20260927 10:48:07.112782 21979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:07.141297 21979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:07.145882 21979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:07.146183 21979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:07.156020 21979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44949
I20260927 10:48:07.156054 22046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44949 every 8 connection(s)
I20260927 10:48:07.157446 21979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:48:07.158794 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 21979
I20260927 10:48:07.159024 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:48:07.159425 392 external_mini_cluster.cc:1428] Setting key 8c6c9cee8c91ce597afe3bd743895f60
I20260927 10:48:07.161693 22047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:07.171301 22047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc46e1a1e7a3499e9eb4f5e02b0bb53a: Bootstrap starting.
Sep 27 10:48:07 dist-test-slave-dzhw krb5kdc[21966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:48:07.174678 22047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc46e1a1e7a3499e9eb4f5e02b0bb53a: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:07.175662 22047 log.cc:824] T 00000000000000000000000000000000 P cc46e1a1e7a3499e9eb4f5e02b0bb53a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:07.178577 22047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc46e1a1e7a3499e9eb4f5e02b0bb53a: No bootstrap required, opened a new log
I20260927 10:48:07.178638 22050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:07.161331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33263 (local address 127.0.98.62:44949)
0927 10:48:07.162053 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:07.162080 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:07.162158 (+ 78us) server_negotiation.cc:407] Connection header received
0927 10:48:07.163065 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:07.163086 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:07.163571 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:07.163937 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:07.165015 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:07.166743 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:07.168027 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:07.168614 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:07.168753 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:07.172669 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:07.172675 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:07.172694 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:07.172737 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:07.174908 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:07.175550 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:07.175557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:07.175563 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:07.175692 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:07.176076 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:07.176079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:07.176080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:07.176746 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:07.176987 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:07.177346 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 10:48:07.177624 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":527,"thread_start_us":303,"threads_started":1}
I20260927 10:48:07.182331 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:48:07.183796 22047 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc46e1a1e7a3499e9eb4f5e02b0bb53a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc46e1a1e7a3499e9eb4f5e02b0bb53a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44949 } }
I20260927 10:48:07.184152 22047 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc46e1a1e7a3499e9eb4f5e02b0bb53a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:07.184273 22047 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc46e1a1e7a3499e9eb4f5e02b0bb53a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc46e1a1e7a3499e9eb4f5e02b0bb53a, State: Initialized, Role: FOLLOWER
I20260927 10:48:07.184892 22047 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc46e1a1e7a3499e9eb4f5e02b0bb53a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc46e1a1e7a3499e9eb4f5e02b0bb53a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44949 } }
I20260927 10:48:07.185235 22047 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc46e1a1e7a3499e9eb4f5e02b0bb53a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:48:07.185340 22047 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc46e1a1e7a3499e9eb4f5e02b0bb53a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:48:07.185488 22047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc46e1a1e7a3499e9eb4f5e02b0bb53a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:07.186968 22047 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc46e1a1e7a3499e9eb4f5e02b0bb53a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc46e1a1e7a3499e9eb4f5e02b0bb53a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44949 } }
I20260927 10:48:07.187489 22047 leader_election.cc:302] T 00000000000000000000000000000000 P cc46e1a1e7a3499e9eb4f5e02b0bb53a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc46e1a1e7a3499e9eb4f5e02b0bb53a; no voters:
I20260927 10:48:07.187991 22047 leader_election.cc:288] T 00000000000000000000000000000000 P cc46e1a1e7a3499e9eb4f5e02b0bb53a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:48:07.188155 22057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc46e1a1e7a3499e9eb4f5e02b0bb53a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:07.188407 22057 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc46e1a1e7a3499e9eb4f5e02b0bb53a [term 1 LEADER]: Becoming Leader. State: Replica: cc46e1a1e7a3499e9eb4f5e02b0bb53a, State: Running, Role: LEADER
I20260927 10:48:07.188903 22057 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc46e1a1e7a3499e9eb4f5e02b0bb53a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc46e1a1e7a3499e9eb4f5e02b0bb53a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44949 } }
I20260927 10:48:07.189976 22047 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc46e1a1e7a3499e9eb4f5e02b0bb53a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:48:07.191210 22058 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc46e1a1e7a3499e9eb4f5e02b0bb53a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc46e1a1e7a3499e9eb4f5e02b0bb53a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc46e1a1e7a3499e9eb4f5e02b0bb53a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44949 } } }
I20260927 10:48:07.191431 22058 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc46e1a1e7a3499e9eb4f5e02b0bb53a [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:07.191916 22058 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc46e1a1e7a3499e9eb4f5e02b0bb53a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc46e1a1e7a3499e9eb4f5e02b0bb53a. Latest consensus state: current_term: 1 leader_uuid: "cc46e1a1e7a3499e9eb4f5e02b0bb53a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc46e1a1e7a3499e9eb4f5e02b0bb53a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44949 } } }
I20260927 10:48:07.192101 22058 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc46e1a1e7a3499e9eb4f5e02b0bb53a [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:07.192515 22066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:48:07.197213 22050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:07.184898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40949 (local address 127.0.98.62:44949)
0927 10:48:07.185056 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:07.185061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:07.185283 (+ 222us) server_negotiation.cc:407] Connection header received
0927 10:48:07.185446 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:07.185449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:07.185513 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:07.185630 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:07.186507 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:07.187430 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:07.188769 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:07.189139 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:07.189255 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:07.191573 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:07.191580 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:07.191584 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:07.191620 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:07.193807 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:07.194533 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:07.194538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:07.194540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:07.194657 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:07.195198 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:07.195203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:07.195204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:07.195575 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:07.195773 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:07.196889 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0927 10:48:07.197022 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:48:07.197636 22066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:48:07.205981 22066 catalog_manager.cc:1347] Generated new cluster ID: b061c1597ead49a799b6926b459b3e3e
I20260927 10:48:07.206187 22066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:48:07.224296 22066 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:48:07.225934 22066 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:48:07.233091 22066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc46e1a1e7a3499e9eb4f5e02b0bb53a: Generated new TSK 0
I20260927 10:48:07.234174 22066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:48:07.240075 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 21979
W20260927 10:48:07.247620 21964 connection.cc:535] client connection to 127.0.98.62:44949 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:44949): Connection reset by peer (error 104)
2026-09-27T10:48:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1526 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:48:07 dist-test-slave-dzhw krb5kdc[22078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:48:07 dist-test-slave-dzhw krb5kdc[22078](info): set up 2 sockets
Sep 27 10:48:07 dist-test-slave-dzhw krb5kdc[22078](info): commencing operation
krb5kdc: starting...
W20260927 10:48:08.319267 392 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:48:08 dist-test-slave-dzhw krb5kdc[22078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:48:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:48:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:48:08.461416 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41517
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:08.601853 22091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:08.602298 22091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:08.602355 22091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:08.607250 22091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:08.607379 22091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:08.607401 22091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:08.607419 22091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:08.607435 22091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:08.614141 22091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41517
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:48:08.615885 22091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:08.617257 22091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:08.628666 22097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:08.628935 22091 server_base.cc:1034] running on GCE node
W20260927 10:48:08.628676 22099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:08.628676 22096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:08.630079 22091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:08.631734 22091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:08.632917 22091 hybrid_clock.cc:648] HybridClock initialized: now 1790506088632893 us; error 50 us; skew 500 ppm
Sep 27 10:48:08 dist-test-slave-dzhw krb5kdc[22078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:08.636407 22091 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:48:08.637619 22091 webserver.cc:460] Webserver started at http://127.0.98.62:42337/ using document root <none> and password file <none>
I20260927 10:48:08.638324 22091 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:08.638396 22091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:08.638731 22091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:08.641436 22091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "db0054e7470f4e4ba73ee715d8739f03"
format_stamp: "Formatted at 2026-09-27 10:48:08 on dist-test-slave-dzhw"
server_key: "74b31a393cfcdd91c13495c6e0c0fd1e"
server_key_iv: "3c4d2be8ac6cfdb9938f477f89ff1af7"
server_key_version: "encryptionkey@0"
I20260927 10:48:08.642125 22091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "db0054e7470f4e4ba73ee715d8739f03"
format_stamp: "Formatted at 2026-09-27 10:48:08 on dist-test-slave-dzhw"
server_key: "74b31a393cfcdd91c13495c6e0c0fd1e"
server_key_iv: "3c4d2be8ac6cfdb9938f477f89ff1af7"
server_key_version: "encryptionkey@0"
I20260927 10:48:08.647104 22091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 10:48:08.650306 22106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:08.652170 22091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:48:08.652390 22091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "db0054e7470f4e4ba73ee715d8739f03"
format_stamp: "Formatted at 2026-09-27 10:48:08 on dist-test-slave-dzhw"
server_key: "74b31a393cfcdd91c13495c6e0c0fd1e"
server_key_iv: "3c4d2be8ac6cfdb9938f477f89ff1af7"
server_key_version: "encryptionkey@0"
I20260927 10:48:08.652573 22091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:08.670025 22091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:08.674180 22091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:08.674504 22091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:08.683281 22091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41517
I20260927 10:48:08.683302 22158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41517 every 8 connection(s)
I20260927 10:48:08.684522 22091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:48:08.688210 22159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:08.691053 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 22091
I20260927 10:48:08.691268 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:48:08.691634 392 external_mini_cluster.cc:1428] Setting key 5e99301316d6f7bbeb1ebfeccaead734
I20260927 10:48:08.696362 22159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db0054e7470f4e4ba73ee715d8739f03: Bootstrap starting.
I20260927 10:48:08.699903 22159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db0054e7470f4e4ba73ee715d8739f03: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:08.701115 22159 log.cc:824] T 00000000000000000000000000000000 P db0054e7470f4e4ba73ee715d8739f03: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:08.703691 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:08.693670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51065 (local address 127.0.98.62:41517)
0927 10:48:08.694295 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:08.694318 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:08.694378 (+ 60us) server_negotiation.cc:407] Connection header received
0927 10:48:08.695364 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:08.695389 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:08.695884 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:08.696292 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:08.696876 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.698137 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.699529 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.700246 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.700414 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:08.701481 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:08.701488 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:08.701511 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:08.702027 (+ 516us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:08.702097 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:08.702124 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:08.702245 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:08.702518 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 10:48:08.702727 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":213,"threads_started":1}
I20260927 10:48:08.703797 22159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db0054e7470f4e4ba73ee715d8739f03: No bootstrap required, opened a new log
I20260927 10:48:08.707646 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:48:08.708935 22159 raft_consensus.cc:348] T 00000000000000000000000000000000 P db0054e7470f4e4ba73ee715d8739f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db0054e7470f4e4ba73ee715d8739f03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41517 } }
I20260927 10:48:08.709331 22159 raft_consensus.cc:374] T 00000000000000000000000000000000 P db0054e7470f4e4ba73ee715d8739f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:08.709416 22159 raft_consensus.cc:729] T 00000000000000000000000000000000 P db0054e7470f4e4ba73ee715d8739f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0054e7470f4e4ba73ee715d8739f03, State: Initialized, Role: FOLLOWER
I20260927 10:48:08.710016 22159 consensus_queue.cc:260] T 00000000000000000000000000000000 P db0054e7470f4e4ba73ee715d8739f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db0054e7470f4e4ba73ee715d8739f03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41517 } }
I20260927 10:48:08.710224 22159 raft_consensus.cc:388] T 00000000000000000000000000000000 P db0054e7470f4e4ba73ee715d8739f03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:48:08.710304 22159 raft_consensus.cc:482] T 00000000000000000000000000000000 P db0054e7470f4e4ba73ee715d8739f03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:48:08.710445 22159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db0054e7470f4e4ba73ee715d8739f03 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:08.711627 22159 raft_consensus.cc:504] T 00000000000000000000000000000000 P db0054e7470f4e4ba73ee715d8739f03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db0054e7470f4e4ba73ee715d8739f03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41517 } }
I20260927 10:48:08.712085 22159 leader_election.cc:302] T 00000000000000000000000000000000 P db0054e7470f4e4ba73ee715d8739f03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db0054e7470f4e4ba73ee715d8739f03; no voters:
I20260927 10:48:08.712500 22159 leader_election.cc:288] T 00000000000000000000000000000000 P db0054e7470f4e4ba73ee715d8739f03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:48:08.712590 22169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db0054e7470f4e4ba73ee715d8739f03 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:08.712865 22169 raft_consensus.cc:686] T 00000000000000000000000000000000 P db0054e7470f4e4ba73ee715d8739f03 [term 1 LEADER]: Becoming Leader. State: Replica: db0054e7470f4e4ba73ee715d8739f03, State: Running, Role: LEADER
I20260927 10:48:08.713289 22169 consensus_queue.cc:237] T 00000000000000000000000000000000 P db0054e7470f4e4ba73ee715d8739f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db0054e7470f4e4ba73ee715d8739f03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41517 } }
I20260927 10:48:08.714305 22159 sys_catalog.cc:564] T 00000000000000000000000000000000 P db0054e7470f4e4ba73ee715d8739f03 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:48:08.715397 22170 sys_catalog.cc:455] T 00000000000000000000000000000000 P db0054e7470f4e4ba73ee715d8739f03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db0054e7470f4e4ba73ee715d8739f03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db0054e7470f4e4ba73ee715d8739f03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41517 } } }
I20260927 10:48:08.715575 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:08.710189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42535 (local address 127.0.98.62:41517)
0927 10:48:08.710334 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:08.710340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:08.710654 (+ 314us) server_negotiation.cc:407] Connection header received
0927 10:48:08.710823 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:08.710827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:08.710887 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:08.710976 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:08.711284 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.711946 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.713074 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.713409 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.713518 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:08.714657 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:08.714661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:08.714664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:08.714679 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:08.714717 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:08.714730 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:08.714834 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:08.715371 (+ 537us) server_negotiation.cc:299] Negotiation successful
0927 10:48:08.715421 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:48:08.715687 22171 sys_catalog.cc:455] T 00000000000000000000000000000000 P db0054e7470f4e4ba73ee715d8739f03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db0054e7470f4e4ba73ee715d8739f03. Latest consensus state: current_term: 1 leader_uuid: "db0054e7470f4e4ba73ee715d8739f03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db0054e7470f4e4ba73ee715d8739f03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41517 } } }
I20260927 10:48:08.716027 22171 sys_catalog.cc:458] T 00000000000000000000000000000000 P db0054e7470f4e4ba73ee715d8739f03 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:08.715801 22170 sys_catalog.cc:458] T 00000000000000000000000000000000 P db0054e7470f4e4ba73ee715d8739f03 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:08.717213 22180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:48:08.720952 22180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:48:08.728463 22180 catalog_manager.cc:1347] Generated new cluster ID: 0ca57b9c611f4f6bad538da0f763d13d
I20260927 10:48:08.728588 22180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:48:08.752161 22180 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:48:08.753803 22180 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:48:08.765844 22180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db0054e7470f4e4ba73ee715d8739f03: Generated new TSK 0
I20260927 10:48:08.767143 22180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:48:08.775970 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 22091
W20260927 10:48:08.782791 22076 connection.cc:535] client connection to 127.0.98.62:41517 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:41517): Connection reset by peer (error 104)
2026-09-27T10:48:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1535 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:48:08 dist-test-slave-dzhw krb5kdc[22190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:48:08 dist-test-slave-dzhw krb5kdc[22190](info): set up 2 sockets
Sep 27 10:48:08 dist-test-slave-dzhw krb5kdc[22190](info): commencing operation
krb5kdc: starting...
W20260927 10:48:09.831429 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:48:09 dist-test-slave-dzhw krb5kdc[22190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:48:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:48:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:48:09.971815 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41613
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:10.105840 22203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:10.106217 22203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:10.106271 22203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:10.110587 22203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:10.110667 22203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:10.110690 22203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:10.110711 22203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:10.110730 22203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:10.116535 22203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41613
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:48:10.117949 22203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:10.119268 22203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:10.129854 22211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:10.129895 22208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:10.129911 22209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:10.130048 22203 server_base.cc:1034] running on GCE node
I20260927 10:48:10.130985 22203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:10.132539 22203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:10.133726 22203 hybrid_clock.cc:648] HybridClock initialized: now 1790506090133702 us; error 38 us; skew 500 ppm
Sep 27 10:48:10 dist-test-slave-dzhw krb5kdc[22190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:10.137184 22203 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:48:10.138612 22203 webserver.cc:460] Webserver started at http://127.0.98.62:41797/ using document root <none> and password file <none>
I20260927 10:48:10.139451 22203 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:10.139562 22203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:10.139914 22203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:10.142985 22203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "591855dba3c44916b58ae16530891718"
format_stamp: "Formatted at 2026-09-27 10:48:10 on dist-test-slave-dzhw"
server_key: "f3fdb36e25736803ff77b7bbe8153f3b"
server_key_iv: "eb69a73973be33bd09dc9097a6b6cd26"
server_key_version: "encryptionkey@0"
I20260927 10:48:10.143733 22203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "591855dba3c44916b58ae16530891718"
format_stamp: "Formatted at 2026-09-27 10:48:10 on dist-test-slave-dzhw"
server_key: "f3fdb36e25736803ff77b7bbe8153f3b"
server_key_iv: "eb69a73973be33bd09dc9097a6b6cd26"
server_key_version: "encryptionkey@0"
I20260927 10:48:10.148810 22203 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 10:48:10.152566 22218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:10.154508 22203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:48:10.154717 22203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "591855dba3c44916b58ae16530891718"
format_stamp: "Formatted at 2026-09-27 10:48:10 on dist-test-slave-dzhw"
server_key: "f3fdb36e25736803ff77b7bbe8153f3b"
server_key_iv: "eb69a73973be33bd09dc9097a6b6cd26"
server_key_version: "encryptionkey@0"
I20260927 10:48:10.154898 22203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:10.161638 22203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:10.165444 22203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:10.165704 22203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:10.175369 22203 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41613
I20260927 10:48:10.175390 22270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41613 every 8 connection(s)
I20260927 10:48:10.176676 22203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:48:10.181175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 22203
I20260927 10:48:10.181380 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:48:10.181635 22271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:10.181847 392 external_mini_cluster.cc:1428] Setting key d9d799440f594229d55d9d91c23f1511
I20260927 10:48:10.190224 22271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 591855dba3c44916b58ae16530891718: Bootstrap starting.
I20260927 10:48:10.190254 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:10.184906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50595 (local address 127.0.98.62:41613)
0927 10:48:10.185529 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:10.185556 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:10.186049 (+ 493us) server_negotiation.cc:407] Connection header received
0927 10:48:10.187004 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:10.187027 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:10.187411 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:10.187844 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:10.188741 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:10.188747 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:10.188772 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:10.188783 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:10.188843 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:10.188873 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:10.189053 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:10.189142 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 10:48:10.189339 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":210,"threads_started":1}
I20260927 10:48:10.193393 22271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 591855dba3c44916b58ae16530891718: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:10.193893 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:48:10.194384 22271 log.cc:824] T 00000000000000000000000000000000 P 591855dba3c44916b58ae16530891718: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:10.197170 22271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 591855dba3c44916b58ae16530891718: No bootstrap required, opened a new log
I20260927 10:48:10.198690 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:10.196550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40583 (local address 127.0.98.62:41613)
0927 10:48:10.196673 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:10.196676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:10.196937 (+ 261us) server_negotiation.cc:407] Connection header received
0927 10:48:10.197102 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:10.197105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:10.197143 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:10.197208 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:10.198090 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:10.198094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:10.198096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:10.198108 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:10.198137 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:10.198148 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:10.198234 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:10.198379 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 10:48:10.198521 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:48:10.201985 22271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 591855dba3c44916b58ae16530891718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591855dba3c44916b58ae16530891718" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } }
I20260927 10:48:10.202292 22271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 591855dba3c44916b58ae16530891718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:10.202327 22271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 591855dba3c44916b58ae16530891718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591855dba3c44916b58ae16530891718, State: Initialized, Role: FOLLOWER
I20260927 10:48:10.202914 22271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 591855dba3c44916b58ae16530891718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591855dba3c44916b58ae16530891718" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } }
I20260927 10:48:10.203056 22271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 591855dba3c44916b58ae16530891718 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:48:10.203155 22271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 591855dba3c44916b58ae16530891718 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:48:10.203284 22271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 591855dba3c44916b58ae16530891718 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:10.204491 22271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 591855dba3c44916b58ae16530891718 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591855dba3c44916b58ae16530891718" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } }
I20260927 10:48:10.204949 22271 leader_election.cc:302] T 00000000000000000000000000000000 P 591855dba3c44916b58ae16530891718 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 591855dba3c44916b58ae16530891718; no voters:
I20260927 10:48:10.205368 22271 leader_election.cc:288] T 00000000000000000000000000000000 P 591855dba3c44916b58ae16530891718 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:48:10.205483 22281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 591855dba3c44916b58ae16530891718 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:10.205768 22281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 591855dba3c44916b58ae16530891718 [term 1 LEADER]: Becoming Leader. State: Replica: 591855dba3c44916b58ae16530891718, State: Running, Role: LEADER
I20260927 10:48:10.206182 22281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 591855dba3c44916b58ae16530891718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591855dba3c44916b58ae16530891718" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } }
I20260927 10:48:10.206995 22271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 591855dba3c44916b58ae16530891718 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:48:10.208290 22282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 591855dba3c44916b58ae16530891718 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "591855dba3c44916b58ae16530891718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591855dba3c44916b58ae16530891718" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } } }
I20260927 10:48:10.208433 22282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 591855dba3c44916b58ae16530891718 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:10.208823 22283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 591855dba3c44916b58ae16530891718 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 591855dba3c44916b58ae16530891718. Latest consensus state: current_term: 1 leader_uuid: "591855dba3c44916b58ae16530891718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "591855dba3c44916b58ae16530891718" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } } }
I20260927 10:48:10.208886 22290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:48:10.208962 22283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 591855dba3c44916b58ae16530891718 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:10.212831 22290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:48:10.220196 22290 catalog_manager.cc:1347] Generated new cluster ID: 82b92135e1a0484a85f11d5f34c0e26c
I20260927 10:48:10.220312 22290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:48:10.231395 22290 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:48:10.231906 22290 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:48:10.238258 22290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 591855dba3c44916b58ae16530891718: Generated new TSK 0
I20260927 10:48:10.239208 22290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:48:10.245268 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 22203
2026-09-27T10:48:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1470 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:48:10 dist-test-slave-dzhw krb5kdc[22302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:48:10 dist-test-slave-dzhw krb5kdc[22302](info): set up 2 sockets
Sep 27 10:48:10 dist-test-slave-dzhw krb5kdc[22302](info): commencing operation
krb5kdc: starting...
W20260927 10:48:11.347502 392 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:48:11 dist-test-slave-dzhw krb5kdc[22302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:48:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:48:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:48:11.490475 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36613
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:11.625660 22315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:11.626102 22315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:11.626187 22315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:11.630682 22315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:11.630788 22315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:11.630829 22315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:11.630887 22315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:11.630928 22315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:11.636934 22315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:48:11.638435 22315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:11.639845 22315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:11.650396 22320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:11.650396 22321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:11.650396 22323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:11.651262 22315 server_base.cc:1034] running on GCE node
I20260927 10:48:11.652395 22315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:11.654160 22315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:11.655400 22315 hybrid_clock.cc:648] HybridClock initialized: now 1790506091655381 us; error 44 us; skew 500 ppm
Sep 27 10:48:11 dist-test-slave-dzhw krb5kdc[22302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:11.668087 22315 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:48:11.669746 22315 webserver.cc:460] Webserver started at http://127.0.98.62:36589/ using document root <none> and password file <none>
I20260927 10:48:11.670797 22315 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:11.670895 22315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:11.671144 22315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:11.673892 22315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "92c66e9ed36b4eb098a08205af2315e2"
format_stamp: "Formatted at 2026-09-27 10:48:11 on dist-test-slave-dzhw"
server_key: "e7556f5b3bbd6740537c0c48dedb85c5"
server_key_iv: "a44bf60396c39c7f42cca8c8c90b0c0b"
server_key_version: "encryptionkey@0"
I20260927 10:48:11.674655 22315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "92c66e9ed36b4eb098a08205af2315e2"
format_stamp: "Formatted at 2026-09-27 10:48:11 on dist-test-slave-dzhw"
server_key: "e7556f5b3bbd6740537c0c48dedb85c5"
server_key_iv: "a44bf60396c39c7f42cca8c8c90b0c0b"
server_key_version: "encryptionkey@0"
I20260927 10:48:11.679730 22315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 10:48:11.684270 22330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:11.686307 22315 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:48:11.686551 22315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "92c66e9ed36b4eb098a08205af2315e2"
format_stamp: "Formatted at 2026-09-27 10:48:11 on dist-test-slave-dzhw"
server_key: "e7556f5b3bbd6740537c0c48dedb85c5"
server_key_iv: "a44bf60396c39c7f42cca8c8c90b0c0b"
server_key_version: "encryptionkey@0"
I20260927 10:48:11.686748 22315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:11.707654 22315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:11.712343 22315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:11.712654 22315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:11.721783 22315 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36613
I20260927 10:48:11.721818 22382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36613 every 8 connection(s)
I20260927 10:48:11.723119 22315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:48:11.726970 22383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:11.731420 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 22315
I20260927 10:48:11.731621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:48:11.731938 392 external_mini_cluster.cc:1428] Setting key cd7f457111974d6a79562662f4f1afef
I20260927 10:48:11.735320 22383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92c66e9ed36b4eb098a08205af2315e2: Bootstrap starting.
I20260927 10:48:11.738370 22383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92c66e9ed36b4eb098a08205af2315e2: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:11.739362 22383 log.cc:824] T 00000000000000000000000000000000 P 92c66e9ed36b4eb098a08205af2315e2: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:11.742355 22383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92c66e9ed36b4eb098a08205af2315e2: No bootstrap required, opened a new log
Sep 27 10:48:11 dist-test-slave-dzhw krb5kdc[22302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:48:11.747756 22383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92c66e9ed36b4eb098a08205af2315e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c66e9ed36b4eb098a08205af2315e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36613 } }
I20260927 10:48:11.748122 22383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92c66e9ed36b4eb098a08205af2315e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:11.748202 22383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92c66e9ed36b4eb098a08205af2315e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92c66e9ed36b4eb098a08205af2315e2, State: Initialized, Role: FOLLOWER
I20260927 10:48:11.748790 22383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92c66e9ed36b4eb098a08205af2315e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c66e9ed36b4eb098a08205af2315e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36613 } }
I20260927 10:48:11.748934 22383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92c66e9ed36b4eb098a08205af2315e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:48:11.749055 22383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92c66e9ed36b4eb098a08205af2315e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:48:11.749200 22383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92c66e9ed36b4eb098a08205af2315e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:11.750663 22383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92c66e9ed36b4eb098a08205af2315e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c66e9ed36b4eb098a08205af2315e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36613 } }
I20260927 10:48:11.751081 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:11.733883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41747 (local address 127.0.98.62:36613)
0927 10:48:11.734607 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:11.734628 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:11.734682 (+ 54us) server_negotiation.cc:407] Connection header received
0927 10:48:11.735564 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:11.735597 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:11.736065 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:11.736500 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:11.737498 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:11.738782 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:11.739763 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:11.740534 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:11.740727 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:11.744042 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:11.744050 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:11.744073 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:11.744112 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:11.747111 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:11.747685 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:11.747691 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:11.747698 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:11.747866 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:11.748286 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:11.748297 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:11.748300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:11.748922 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:11.749189 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:11.749499 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 10:48:11.749925 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":272,"threads_started":1}
I20260927 10:48:11.751340 22383 leader_election.cc:302] T 00000000000000000000000000000000 P 92c66e9ed36b4eb098a08205af2315e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92c66e9ed36b4eb098a08205af2315e2; no voters:
I20260927 10:48:11.751780 22383 leader_election.cc:288] T 00000000000000000000000000000000 P 92c66e9ed36b4eb098a08205af2315e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:48:11.751955 22388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92c66e9ed36b4eb098a08205af2315e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:11.752218 22388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92c66e9ed36b4eb098a08205af2315e2 [term 1 LEADER]: Becoming Leader. State: Replica: 92c66e9ed36b4eb098a08205af2315e2, State: Running, Role: LEADER
I20260927 10:48:11.752705 22388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92c66e9ed36b4eb098a08205af2315e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c66e9ed36b4eb098a08205af2315e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36613 } }
I20260927 10:48:11.753360 22383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92c66e9ed36b4eb098a08205af2315e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:48:11.754788 22390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c66e9ed36b4eb098a08205af2315e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92c66e9ed36b4eb098a08205af2315e2. Latest consensus state: current_term: 1 leader_uuid: "92c66e9ed36b4eb098a08205af2315e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c66e9ed36b4eb098a08205af2315e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36613 } } }
I20260927 10:48:11.754963 22389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c66e9ed36b4eb098a08205af2315e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92c66e9ed36b4eb098a08205af2315e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c66e9ed36b4eb098a08205af2315e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36613 } } }
I20260927 10:48:11.754966 22390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c66e9ed36b4eb098a08205af2315e2 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:11.755051 22389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c66e9ed36b4eb098a08205af2315e2 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:11.755090 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:48:11.755501 22397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:48:11.759799 22397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:48:11.769028 22397 catalog_manager.cc:1347] Generated new cluster ID: 5397d4afb1fd4fa98b9192819470edbe
I20260927 10:48:11.769135 22397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:48:11.775701 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:11.758308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50767 (local address 127.0.98.62:36613)
0927 10:48:11.758470 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:11.758475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:11.758980 (+ 505us) server_negotiation.cc:407] Connection header received
0927 10:48:11.759149 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:11.759155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:11.759221 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:11.759298 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:11.760423 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:11.761122 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:11.762057 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:11.762341 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:11.762514 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:11.764536 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:11.764543 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:11.764549 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:11.764594 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:11.767250 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:11.767961 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:11.767966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:11.767968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:11.768089 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:11.768478 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:11.768482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:11.768484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:11.768783 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:11.768940 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:11.775241 (+ 6301us) server_negotiation.cc:299] Negotiation successful
0927 10:48:11.775461 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 10:48:11.782998 22397 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:48:11.784745 22397 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:48:11.791973 22397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92c66e9ed36b4eb098a08205af2315e2: Generated new TSK 0
I20260927 10:48:11.793131 22397 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:48:11.801007 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 22315
2026-09-27T10:48:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1554 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:48:11 dist-test-slave-dzhw krb5kdc[22414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:48:11 dist-test-slave-dzhw krb5kdc[22414](info): set up 2 sockets
Sep 27 10:48:11 dist-test-slave-dzhw krb5kdc[22414](info): commencing operation
krb5kdc: starting...
W20260927 10:48:12.867303 392 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:48:12 dist-test-slave-dzhw krb5kdc[22414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:48:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:48:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:48:13.004205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38971
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:13.135396 22427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:13.135778 22427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:13.135828 22427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:13.140306 22427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:13.140425 22427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:13.140447 22427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:13.140466 22427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:13.140484 22427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:13.146812 22427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:48:13.148334 22427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:13.149691 22427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:13.159940 22432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:13.160079 22427 server_base.cc:1034] running on GCE node
W20260927 10:48:13.159935 22433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:13.159940 22435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:13.160902 22427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:13.162338 22427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:13.163578 22427 hybrid_clock.cc:648] HybridClock initialized: now 1790506093163564 us; error 32 us; skew 500 ppm
Sep 27 10:48:13 dist-test-slave-dzhw krb5kdc[22414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:13.166894 22427 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:48:13.168119 22427 webserver.cc:460] Webserver started at http://127.0.98.62:34959/ using document root <none> and password file <none>
I20260927 10:48:13.168896 22427 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:13.169014 22427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:13.169324 22427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:13.172010 22427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "594fe7c55349424285b78451b3412633"
format_stamp: "Formatted at 2026-09-27 10:48:13 on dist-test-slave-dzhw"
server_key: "ab784b838780564b67ddd0260b654223"
server_key_iv: "2f62bcd0abcd8ed2fec787fdce96da90"
server_key_version: "encryptionkey@0"
I20260927 10:48:13.172683 22427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "594fe7c55349424285b78451b3412633"
format_stamp: "Formatted at 2026-09-27 10:48:13 on dist-test-slave-dzhw"
server_key: "ab784b838780564b67ddd0260b654223"
server_key_iv: "2f62bcd0abcd8ed2fec787fdce96da90"
server_key_version: "encryptionkey@0"
I20260927 10:48:13.177284 22427 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 10:48:13.180408 22442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:13.181948 22427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:48:13.182124 22427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "594fe7c55349424285b78451b3412633"
format_stamp: "Formatted at 2026-09-27 10:48:13 on dist-test-slave-dzhw"
server_key: "ab784b838780564b67ddd0260b654223"
server_key_iv: "2f62bcd0abcd8ed2fec787fdce96da90"
server_key_version: "encryptionkey@0"
I20260927 10:48:13.182266 22427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:13.199219 22427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:13.202966 22427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:13.203214 22427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:13.212178 22427 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38971
I20260927 10:48:13.212208 22494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38971 every 8 connection(s)
I20260927 10:48:13.213338 22427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:48:13.216667 22495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:13.223136 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 22427
I20260927 10:48:13.223294 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:48:13.223469 22495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594fe7c55349424285b78451b3412633: Bootstrap starting.
I20260927 10:48:13.223626 392 external_mini_cluster.cc:1428] Setting key 815261a9adaa7c614df7fa0c214f6809
I20260927 10:48:13.226615 22495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 594fe7c55349424285b78451b3412633: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:13.227495 22495 log.cc:824] T 00000000000000000000000000000000 P 594fe7c55349424285b78451b3412633: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:13.229614 22495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594fe7c55349424285b78451b3412633: No bootstrap required, opened a new log
I20260927 10:48:13.233880 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:13.225287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32823 (local address 127.0.98.62:38971)
0927 10:48:13.225818 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:13.225837 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:13.225877 (+ 40us) server_negotiation.cc:407] Connection header received
0927 10:48:13.226699 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:13.226741 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:13.227182 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:13.227612 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:13.228076 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:13.229133 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:13.230147 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:13.230736 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:13.230917 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:13.231889 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:13.231895 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:13.231909 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:13.232364 (+ 455us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:13.232450 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:13.232477 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:13.232598 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:13.232715 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 10:48:13.232969 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":147,"threads_started":1}
I20260927 10:48:13.233716 22495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 594fe7c55349424285b78451b3412633 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594fe7c55349424285b78451b3412633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38971 } }
I20260927 10:48:13.233999 22495 raft_consensus.cc:374] T 00000000000000000000000000000000 P 594fe7c55349424285b78451b3412633 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:13.234059 22495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 594fe7c55349424285b78451b3412633 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 594fe7c55349424285b78451b3412633, State: Initialized, Role: FOLLOWER
I20260927 10:48:13.234632 22495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 594fe7c55349424285b78451b3412633 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594fe7c55349424285b78451b3412633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38971 } }
I20260927 10:48:13.234798 22495 raft_consensus.cc:388] T 00000000000000000000000000000000 P 594fe7c55349424285b78451b3412633 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:48:13.234874 22495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 594fe7c55349424285b78451b3412633 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:48:13.234985 22495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 594fe7c55349424285b78451b3412633 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:13.236356 22495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 594fe7c55349424285b78451b3412633 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594fe7c55349424285b78451b3412633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38971 } }
I20260927 10:48:13.236793 22495 leader_election.cc:302] T 00000000000000000000000000000000 P 594fe7c55349424285b78451b3412633 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 594fe7c55349424285b78451b3412633; no voters:
I20260927 10:48:13.237159 22495 leader_election.cc:288] T 00000000000000000000000000000000 P 594fe7c55349424285b78451b3412633 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:48:13.237291 22500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 594fe7c55349424285b78451b3412633 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:13.237506 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:48:13.237599 22500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 594fe7c55349424285b78451b3412633 [term 1 LEADER]: Becoming Leader. State: Replica: 594fe7c55349424285b78451b3412633, State: Running, Role: LEADER
I20260927 10:48:13.237926 22500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 594fe7c55349424285b78451b3412633 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594fe7c55349424285b78451b3412633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38971 } }
I20260927 10:48:13.238698 22495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 594fe7c55349424285b78451b3412633 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:48:13.239892 22501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594fe7c55349424285b78451b3412633 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "594fe7c55349424285b78451b3412633" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594fe7c55349424285b78451b3412633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38971 } } }
I20260927 10:48:13.240063 22501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594fe7c55349424285b78451b3412633 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:13.240296 22501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594fe7c55349424285b78451b3412633 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 594fe7c55349424285b78451b3412633. Latest consensus state: current_term: 1 leader_uuid: "594fe7c55349424285b78451b3412633" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594fe7c55349424285b78451b3412633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38971 } } }
I20260927 10:48:13.240391 22501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594fe7c55349424285b78451b3412633 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:13.240808 22513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:48:13.244412 22513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:48:13.247509 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:13.240184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33201 (local address 127.0.98.62:38971)
0927 10:48:13.240315 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:13.240320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:13.242818 (+ 2498us) server_negotiation.cc:407] Connection header received
0927 10:48:13.243031 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:13.243035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:13.243091 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:13.243147 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:13.243602 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:13.244362 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:13.245536 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:13.245869 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:13.245975 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:13.247021 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:13.247025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:13.247028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:13.247042 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:13.247080 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:13.247091 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:13.247172 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:13.247294 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 10:48:13.247354 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:48:13.251127 22513 catalog_manager.cc:1347] Generated new cluster ID: b50a2ea95ca145649c223accbe49d6a4
I20260927 10:48:13.251194 22513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:48:13.269788 22513 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:48:13.271307 22513 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:48:13.278703 22513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 594fe7c55349424285b78451b3412633: Generated new TSK 0
I20260927 10:48:13.279805 22513 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:48:13.288237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 22427
2026-09-27T10:48:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1486 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:48:13 dist-test-slave-dzhw krb5kdc[22526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:48:13 dist-test-slave-dzhw krb5kdc[22526](info): set up 2 sockets
Sep 27 10:48:13 dist-test-slave-dzhw krb5kdc[22526](info): commencing operation
krb5kdc: starting...
W20260927 10:48:14.363255 392 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:48:14 dist-test-slave-dzhw krb5kdc[22526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:48:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:48:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:48:14.510289 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39611
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:14.643651 22539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:14.643970 22539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:14.644018 22539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:14.648248 22539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:14.648324 22539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:14.648351 22539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:14.648370 22539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:14.648386 22539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:14.653977 22539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:48:14.655645 22539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:14.656984 22539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:14.667838 22545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:14.667806 22547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:14.668026 22539 server_base.cc:1034] running on GCE node
W20260927 10:48:14.667845 22544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:14.669111 22539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:14.670815 22539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:14.672050 22539 hybrid_clock.cc:648] HybridClock initialized: now 1790506094672027 us; error 45 us; skew 500 ppm
Sep 27 10:48:14 dist-test-slave-dzhw krb5kdc[22526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:14.675835 22539 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:48:14.677155 22539 webserver.cc:460] Webserver started at http://127.0.98.62:46085/ using document root <none> and password file <none>
I20260927 10:48:14.677934 22539 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:14.678045 22539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:14.678375 22539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:14.681150 22539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "f6cc85951d3b41009fa5921999198fd0"
format_stamp: "Formatted at 2026-09-27 10:48:14 on dist-test-slave-dzhw"
server_key: "1abe5fad5f05fb148ed2620be8096933"
server_key_iv: "f065927113618919a0ad22d84dd37d55"
server_key_version: "encryptionkey@0"
I20260927 10:48:14.681866 22539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "f6cc85951d3b41009fa5921999198fd0"
format_stamp: "Formatted at 2026-09-27 10:48:14 on dist-test-slave-dzhw"
server_key: "1abe5fad5f05fb148ed2620be8096933"
server_key_iv: "f065927113618919a0ad22d84dd37d55"
server_key_version: "encryptionkey@0"
I20260927 10:48:14.687106 22539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260927 10:48:14.690845 22554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:14.692736 22539 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:48:14.692935 22539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "f6cc85951d3b41009fa5921999198fd0"
format_stamp: "Formatted at 2026-09-27 10:48:14 on dist-test-slave-dzhw"
server_key: "1abe5fad5f05fb148ed2620be8096933"
server_key_iv: "f065927113618919a0ad22d84dd37d55"
server_key_version: "encryptionkey@0"
I20260927 10:48:14.693106 22539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:14.699223 22539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:14.703251 22539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:14.703513 22539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:14.712931 22539 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39611
I20260927 10:48:14.713008 22606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39611 every 8 connection(s)
I20260927 10:48:14.714421 22539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:48:14.718206 22607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:14.719245 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 22539
I20260927 10:48:14.719404 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:48:14.719758 392 external_mini_cluster.cc:1428] Setting key 30947587752fd13ea4f84821c2234319
I20260927 10:48:14.725973 22607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6cc85951d3b41009fa5921999198fd0: Bootstrap starting.
I20260927 10:48:14.726581 22610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:14.721483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42301 (local address 127.0.98.62:39611)
0927 10:48:14.722034 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:14.722052 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:14.722097 (+ 45us) server_negotiation.cc:407] Connection header received
0927 10:48:14.723027 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:14.723052 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:14.723425 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:14.723780 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:14.724809 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:14.724815 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:14.724830 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:14.724841 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:14.724933 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:14.724957 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:14.725124 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:14.725246 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 10:48:14.725474 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":176,"threads_started":1}
I20260927 10:48:14.728971 22607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6cc85951d3b41009fa5921999198fd0: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:14.729892 22607 log.cc:824] T 00000000000000000000000000000000 P f6cc85951d3b41009fa5921999198fd0: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:14.730576 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:48:14.732959 22607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6cc85951d3b41009fa5921999198fd0: No bootstrap required, opened a new log
I20260927 10:48:14.735419 22610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:14.733259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52917 (local address 127.0.98.62:39611)
0927 10:48:14.733437 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:14.733443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:14.733740 (+ 297us) server_negotiation.cc:407] Connection header received
0927 10:48:14.733894 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:14.733896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:14.733948 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:14.734034 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:14.734911 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:14.734914 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:14.734916 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:14.734926 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:14.734970 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:14.734982 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:14.735060 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:14.735178 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 10:48:14.735227 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 10:48:14.738024 22607 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6cc85951d3b41009fa5921999198fd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cc85951d3b41009fa5921999198fd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39611 } }
I20260927 10:48:14.738390 22607 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6cc85951d3b41009fa5921999198fd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:14.738487 22607 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6cc85951d3b41009fa5921999198fd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6cc85951d3b41009fa5921999198fd0, State: Initialized, Role: FOLLOWER
I20260927 10:48:14.739022 22607 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6cc85951d3b41009fa5921999198fd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cc85951d3b41009fa5921999198fd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39611 } }
I20260927 10:48:14.739161 22607 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6cc85951d3b41009fa5921999198fd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:48:14.739217 22607 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6cc85951d3b41009fa5921999198fd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:48:14.739300 22607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6cc85951d3b41009fa5921999198fd0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:14.740228 22607 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6cc85951d3b41009fa5921999198fd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cc85951d3b41009fa5921999198fd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39611 } }
I20260927 10:48:14.740558 22607 leader_election.cc:302] T 00000000000000000000000000000000 P f6cc85951d3b41009fa5921999198fd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6cc85951d3b41009fa5921999198fd0; no voters:
I20260927 10:48:14.740898 22607 leader_election.cc:288] T 00000000000000000000000000000000 P f6cc85951d3b41009fa5921999198fd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:48:14.741073 22617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6cc85951d3b41009fa5921999198fd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:14.741407 22617 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6cc85951d3b41009fa5921999198fd0 [term 1 LEADER]: Becoming Leader. State: Replica: f6cc85951d3b41009fa5921999198fd0, State: Running, Role: LEADER
I20260927 10:48:14.741904 22617 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6cc85951d3b41009fa5921999198fd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cc85951d3b41009fa5921999198fd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39611 } }
I20260927 10:48:14.742491 22607 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6cc85951d3b41009fa5921999198fd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:48:14.744069 22618 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6cc85951d3b41009fa5921999198fd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6cc85951d3b41009fa5921999198fd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cc85951d3b41009fa5921999198fd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39611 } } }
I20260927 10:48:14.744251 22618 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6cc85951d3b41009fa5921999198fd0 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:14.744616 22619 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6cc85951d3b41009fa5921999198fd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6cc85951d3b41009fa5921999198fd0. Latest consensus state: current_term: 1 leader_uuid: "f6cc85951d3b41009fa5921999198fd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cc85951d3b41009fa5921999198fd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39611 } } }
I20260927 10:48:14.744711 22619 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6cc85951d3b41009fa5921999198fd0 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:14.744930 22629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:48:14.749203 22629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:48:14.756585 22629 catalog_manager.cc:1347] Generated new cluster ID: 295341d8068a42a88fc99ff9d201eb65
I20260927 10:48:14.756691 22629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:48:14.767719 22629 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:48:14.768230 22629 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:48:14.777408 22629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6cc85951d3b41009fa5921999198fd0: Generated new TSK 0
I20260927 10:48:14.778517 22629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:48:14.788945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 22539
2026-09-27T10:48:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1500 ms)
[----------] 18 tests from AuthTokenIssuingTest (26950 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:48:14 dist-test-slave-dzhw krb5kdc[22638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:48:14 dist-test-slave-dzhw krb5kdc[22638](info): set up 2 sockets
Sep 27 10:48:14 dist-test-slave-dzhw krb5kdc[22638](info): commencing operation
krb5kdc: starting...
W20260927 10:48:15.843336 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:48:15 dist-test-slave-dzhw krb5kdc[22638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:48:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:48:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:48:15.981631 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33301
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33301,127.0.98.61:33257,127.0.98.60:32885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:16.117902 22651 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:48:16.118266 22651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:16.118292 22651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:16.118394 22651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:16.122668 22651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:16.122743 22651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:16.122763 22651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:16.122781 22651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:16.122829 22651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:16.128743 22651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33301,127.0.98.61:33257,127.0.98.60:32885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:48:16.130173 22651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:16.131493 22651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:16.142040 22656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:16.142242 22651 server_base.cc:1034] running on GCE node
W20260927 10:48:16.142038 22659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:16.142077 22657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:16.143111 22651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:16.144469 22651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:16.145643 22651 hybrid_clock.cc:648] HybridClock initialized: now 1790506096145614 us; error 46 us; skew 500 ppm
Sep 27 10:48:16 dist-test-slave-dzhw krb5kdc[22638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:16.148820 22651 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:48:16.149904 22651 webserver.cc:460] Webserver started at http://127.0.98.62:44189/ using document root <none> and password file <none>
I20260927 10:48:16.150647 22651 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:16.150733 22651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:16.150949 22651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:16.153487 22651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "49a02de860cc4112afdb4de6f3e149fd"
format_stamp: "Formatted at 2026-09-27 10:48:16 on dist-test-slave-dzhw"
server_key: "41c4037e9973c8e34789f653578d607c"
server_key_iv: "d0747490e4787497612b668b9c0ffbed"
server_key_version: "encryptionkey@0"
I20260927 10:48:16.154138 22651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "49a02de860cc4112afdb4de6f3e149fd"
format_stamp: "Formatted at 2026-09-27 10:48:16 on dist-test-slave-dzhw"
server_key: "41c4037e9973c8e34789f653578d607c"
server_key_iv: "d0747490e4787497612b668b9c0ffbed"
server_key_version: "encryptionkey@0"
I20260927 10:48:16.158871 22651 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260927 10:48:16.162284 22666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:16.164495 22651 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 10:48:16.164713 22651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "49a02de860cc4112afdb4de6f3e149fd"
format_stamp: "Formatted at 2026-09-27 10:48:16 on dist-test-slave-dzhw"
server_key: "41c4037e9973c8e34789f653578d607c"
server_key_iv: "d0747490e4787497612b668b9c0ffbed"
server_key_version: "encryptionkey@0"
I20260927 10:48:16.164901 22651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:16.191658 22651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:16.195720 22651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:16.195967 22651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:16.204826 22651 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33301
I20260927 10:48:16.205199 22718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33301 every 8 connection(s)
I20260927 10:48:16.206159 22651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:48:16.210028 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 22651
I20260927 10:48:16.210156 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:48:16.210531 22719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:16.210580 392 external_mini_cluster.cc:1428] Setting key 6bee2954b359e2c96da3dc797da74a56
I20260927 10:48:16.215875 22719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 10:48:16 dist-test-slave-dzhw krb5kdc[22638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:48:16.234793 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.218928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59104 (local address 127.0.98.62:33301)
0927 10:48:16.219636 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:16.219659 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:16.219801 (+ 142us) server_negotiation.cc:407] Connection header received
0927 10:48:16.221293 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:16.221298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:16.221533 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:16.221716 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:16.222965 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.223922 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.225251 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.225803 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.225943 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:16.229136 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:16.229143 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:16.229156 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:16.229191 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:16.231144 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.231661 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.231672 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.231673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.231778 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.232100 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.232103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.232113 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.232565 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:16.232723 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:16.233148 (+ 425us) server_negotiation.cc:299] Negotiation successful
0927 10:48:16.233457 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"spinlock_wait_cycles":82176,"thread_start_us":178,"threads_started":1}
I20260927 10:48:16.234793 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.218928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33301 (local address 127.0.0.1:59104)
0927 10:48:16.219592 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.219652 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:16.220952 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:16.221776 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:16.221782 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:16.222300 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:16.222735 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:16.222758 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.224070 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.224078 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:16.225117 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:16.225127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.225969 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.225973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:16.226075 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:16.226863 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:16.226884 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:16.228930 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:16.231281 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:16.231293 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:16.231304 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:16.231537 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:16.231879 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:16.231884 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:16.231886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:16.232001 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:16.232687 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:16.232695 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:16.232881 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 10:48:16.233066 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 10:48:16.233526 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":114,"thread_start_us":201,"threads_started":1}
I20260927 10:48:16.237952 22719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33257 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:48:16.239120 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.238693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33257 (local address 127.0.0.1:49686)
0927 10:48:16.238877 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.238989 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260927 10:48:16.239425 22667 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33257: connect: Connection refused (error 111)
I20260927 10:48:16.240522 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.240266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33257 (local address 127.0.0.1:49694)
0927 10:48:16.240395 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.240452 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 10:48:16.240720 22719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33257: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33257: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 10:48:16.242690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33257
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33301,127.0.98.61:33257,127.0.98.60:32885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:48:16.290097 22719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33257 } attempt: 1
I20260927 10:48:16.291507 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.291034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33257 (local address 127.0.0.1:49708)
0927 10:48:16.291275 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.291399 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 10:48:16.292397 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.292155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33257 (local address 127.0.0.1:49710)
0927 10:48:16.292283 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.292335 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 10:48:16.292562 22719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33257: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33257: connect: Connection refused (error 111)
I20260927 10:48:16.360793 22719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33257 } attempt: 2
I20260927 10:48:16.361965 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.361573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33257 (local address 127.0.0.1:49720)
0927 10:48:16.361784 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.361866 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 10:48:16.362941 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.362714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33257 (local address 127.0.0.1:49724)
0927 10:48:16.362831 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.362879 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 10:48:16.363093 22719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33257: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33257: connect: Connection refused (error 111)
W20260927 10:48:16.375846 22726 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:48:16.376194 22726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:16.376221 22726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:16.376264 22726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:16.380518 22726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:16.380582 22726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:16.380604 22726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:16.380621 22726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:16.380636 22726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:16.386302 22726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33301,127.0.98.61:33257,127.0.98.60:32885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:48:16.387722 22726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:16.388960 22726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:16.399121 22731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:16.399186 22734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:16.399242 22726 server_base.cc:1034] running on GCE node
W20260927 10:48:16.399152 22732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:16.400177 22726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:16.401538 22726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:16.402786 22726 hybrid_clock.cc:648] HybridClock initialized: now 1790506096402773 us; error 34 us; skew 500 ppm
Sep 27 10:48:16 dist-test-slave-dzhw krb5kdc[22638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:16.406246 22726 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 10:48:16.407516 22726 webserver.cc:460] Webserver started at http://127.0.98.61:42083/ using document root <none> and password file <none>
I20260927 10:48:16.408237 22726 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:16.408349 22726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:16.408636 22726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:16.411427 22726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-1/data/instance:
uuid: "8e0540b0e4d645159229541979245acc"
format_stamp: "Formatted at 2026-09-27 10:48:16 on dist-test-slave-dzhw"
server_key: "b2b6a97beff371d3a6691fba9879573a"
server_key_iv: "9db7fdd6960b9d6566e5b4c505a91d23"
server_key_version: "encryptionkey@0"
I20260927 10:48:16.412122 22726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-1/wal/instance:
uuid: "8e0540b0e4d645159229541979245acc"
format_stamp: "Formatted at 2026-09-27 10:48:16 on dist-test-slave-dzhw"
server_key: "b2b6a97beff371d3a6691fba9879573a"
server_key_iv: "9db7fdd6960b9d6566e5b4c505a91d23"
server_key_version: "encryptionkey@0"
I20260927 10:48:16.416878 22726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:48:16.420125 22741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:16.421868 22726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 10:48:16.422068 22726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-1/wal
uuid: "8e0540b0e4d645159229541979245acc"
format_stamp: "Formatted at 2026-09-27 10:48:16 on dist-test-slave-dzhw"
server_key: "b2b6a97beff371d3a6691fba9879573a"
server_key_iv: "9db7fdd6960b9d6566e5b4c505a91d23"
server_key_version: "encryptionkey@0"
I20260927 10:48:16.422235 22726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:16.436576 22726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:16.440506 22726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:16.440781 22726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:16.449718 22726 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33257
I20260927 10:48:16.450078 22793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33257 every 8 connection(s)
I20260927 10:48:16.451033 22726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-1/data/info.pb
I20260927 10:48:16.454303 22719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33257 } attempt: 3
I20260927 10:48:16.454819 22794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:16.460173 22794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:48:16.461280 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 22726
I20260927 10:48:16.461433 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-1/wal/instance
I20260927 10:48:16.461786 392 external_mini_cluster.cc:1428] Setting key 989c8351c5d95bf98c433590b2537d10
Sep 27 10:48:16 dist-test-slave-dzhw krb5kdc[22638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 10:48:16 dist-test-slave-dzhw krb5kdc[22638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:48:16.469617 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.455087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33257 (local address 127.0.0.1:49734)
0927 10:48:16.455274 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.455296 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:16.455412 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:16.457950 (+ 2538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:16.457955 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:16.458217 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:16.458390 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:16.458399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.459828 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.459832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:16.460684 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:16.460697 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.461678 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.461682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:16.461796 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:16.462471 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:16.462485 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:16.464265 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:16.466860 (+ 2595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:16.466866 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:16.466872 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:16.467182 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:16.467955 (+ 773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:16.467961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:16.467964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:16.468078 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:16.468983 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:16.468987 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:16.469099 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 10:48:16.469221 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 10:48:16.469395 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 10:48:16.471850 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.455863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49734 (local address 127.0.98.61:33257)
0927 10:48:16.456339 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:16.456361 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:16.456412 (+ 51us) server_negotiation.cc:407] Connection header received
0927 10:48:16.457208 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:16.457227 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:16.457606 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:16.457951 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:16.458583 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.459688 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.460803 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.461546 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.461718 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:16.464472 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:16.464480 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:16.464496 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:16.464533 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:16.466740 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.467319 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.467325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.467333 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.467496 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.468196 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.468200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.468202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.468848 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:16.469010 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:16.469427 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 10:48:16.469824 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":163,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 10:48:16.473628 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.462480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59106 (local address 127.0.98.62:33301)
0927 10:48:16.462633 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:16.462637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:16.463073 (+ 436us) server_negotiation.cc:407] Connection header received
0927 10:48:16.463325 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:16.463331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:16.463392 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:16.463488 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:16.464566 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.465323 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.466718 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.467054 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.467175 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:16.469868 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:16.469873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:16.469875 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:16.469899 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:16.471562 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.472227 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.472230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.472232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.472318 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.472591 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.472595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.472597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.472789 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:16.472903 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:16.473323 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 10:48:16.473458 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 10:48:16.473661 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.462463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33301 (local address 127.0.0.1:59106)
0927 10:48:16.462924 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.462984 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:16.463184 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:16.463578 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:16.463584 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:16.464096 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:16.464312 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:16.464322 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.465447 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.465451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:16.466580 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:16.466595 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.467176 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.467179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:16.467302 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:16.467896 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:16.467914 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:16.469711 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:16.471681 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:16.471692 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:16.471718 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:16.472113 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:16.472422 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:16.472424 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:16.472425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:16.472497 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:16.472905 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:16.472913 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:16.473051 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 10:48:16.473248 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 10:48:16.473467 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":107,"threads_started":1}
I20260927 10:48:16.474680 22719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:48:16.475543 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.475261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32885 (local address 127.0.0.1:57256)
0927 10:48:16.475426 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.475481 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260927 10:48:16.476351 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.476060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32885 (local address 127.0.0.1:57266)
0927 10:48:16.476210 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.476289 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 10:48:16.476456 22794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33257 } has no permanent_uuid. Determining permanent_uuid...
W20260927 10:48:16.476502 22719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32885: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32885: connect: Connection refused (error 111)
Sep 27 10:48:16 dist-test-slave-dzhw krb5kdc[22638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 10:48:16.485783 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.476944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33257 (local address 127.0.0.1:49738)
0927 10:48:16.477057 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.477072 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:16.477155 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:16.477480 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:16.477482 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:16.477628 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:16.477724 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:16.477729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.478838 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.478842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:16.479762 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:16.479774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.480207 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.480214 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:16.480344 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:16.480849 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:16.480893 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:16.482514 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:16.484279 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:16.484284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:16.484286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:16.484486 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:16.484758 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:16.484763 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:16.484765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:16.484847 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:16.485232 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:16.485235 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:16.485325 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 10:48:16.485444 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 10:48:16.485593 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 10:48:16.485783 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.477096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49738 (local address 127.0.98.61:33257)
0927 10:48:16.477232 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:16.477238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:16.477253 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:48:16.477293 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:16.477298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:16.477354 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:16.477458 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:16.477832 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.478709 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.479873 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.480096 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.480173 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:16.482648 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:16.482651 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:16.482653 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:16.482673 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:16.484148 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.484580 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.484586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.484588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.484661 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.484935 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.484939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.484941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.485131 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:16.485234 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:16.485487 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 10:48:16.485650 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":45}
I20260927 10:48:16.486804 22794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:48:16.487633 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.487328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32885 (local address 127.0.0.1:57278)
0927 10:48:16.487459 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.487570 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 10:48:16.488286 22743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32885: connect: Connection refused (error 111)
I20260927 10:48:16.489305 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.489065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32885 (local address 127.0.0.1:57280)
0927 10:48:16.489177 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.489232 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 10:48:16.489444 22794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32885: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32885: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 10:48:16.494706 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:32885
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33301,127.0.98.61:33257,127.0.98.60:32885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:48:16.507692 22719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } attempt: 1
I20260927 10:48:16.508786 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.508421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32885 (local address 127.0.0.1:57286)
0927 10:48:16.508606 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.508684 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 10:48:16.509639 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.509380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32885 (local address 127.0.0.1:57290)
0927 10:48:16.509499 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.509575 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 10:48:16.509774 22719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32885: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32885: connect: Connection refused (error 111)
I20260927 10:48:16.538738 22794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } attempt: 1
I20260927 10:48:16.539880 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.539469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32885 (local address 127.0.0.1:57304)
0927 10:48:16.539694 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.539777 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260927 10:48:16.540648 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.540430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32885 (local address 127.0.0.1:57312)
0927 10:48:16.540539 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.540593 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 10:48:16.540771 22794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32885: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32885: connect: Connection refused (error 111)
I20260927 10:48:16.584968 22719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } attempt: 2
I20260927 10:48:16.586158 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.585760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32885 (local address 127.0.0.1:57318)
0927 10:48:16.585965 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.586051 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 10:48:16.587034 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.586799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32885 (local address 127.0.0.1:57332)
0927 10:48:16.586917 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.586974 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 10:48:16.587178 22719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32885: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32885: connect: Connection refused (error 111)
I20260927 10:48:16.608948 22794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } attempt: 2
I20260927 10:48:16.610132 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.609684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32885 (local address 127.0.0.1:57344)
0927 10:48:16.609911 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.610008 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 10:48:16.611021 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.610735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32885 (local address 127.0.0.1:57360)
0927 10:48:16.610888 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.610944 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 10:48:16.611163 22794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32885: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32885: connect: Connection refused (error 111)
W20260927 10:48:16.632139 22801 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:48:16.632511 22801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:16.632576 22801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:16.632653 22801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:16.637003 22801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:16.637089 22801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:16.637108 22801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:16.637151 22801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:16.637187 22801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:16.643006 22801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33301,127.0.98.61:33257,127.0.98.60:32885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:32885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:48:16.644423 22801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:16.645766 22801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:16.655982 22806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:16.656002 22807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:16.656122 22801 server_base.cc:1034] running on GCE node
W20260927 10:48:16.656314 22809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:16.657104 22801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:16.658548 22801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:16.659744 22801 hybrid_clock.cc:648] HybridClock initialized: now 1790506096659719 us; error 36 us; skew 500 ppm
Sep 27 10:48:16 dist-test-slave-dzhw krb5kdc[22638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:16.662995 22801 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 10:48:16.664076 22801 webserver.cc:460] Webserver started at http://127.0.98.60:46441/ using document root <none> and password file <none>
I20260927 10:48:16.664778 22801 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:16.664844 22801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:16.665120 22801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:16.667747 22801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-2/data/instance:
uuid: "7c47bd901a35409e9687e6c2cd39191a"
format_stamp: "Formatted at 2026-09-27 10:48:16 on dist-test-slave-dzhw"
server_key: "a890daca5179e7abb1f2367ce4f6f734"
server_key_iv: "eb9530b7d4599cca015f6c056fe08752"
server_key_version: "encryptionkey@0"
I20260927 10:48:16.668432 22801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-2/wal/instance:
uuid: "7c47bd901a35409e9687e6c2cd39191a"
format_stamp: "Formatted at 2026-09-27 10:48:16 on dist-test-slave-dzhw"
server_key: "a890daca5179e7abb1f2367ce4f6f734"
server_key_iv: "eb9530b7d4599cca015f6c056fe08752"
server_key_version: "encryptionkey@0"
I20260927 10:48:16.673395 22801 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 10:48:16.676609 22816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:16.678251 22801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:48:16.678452 22801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-2/wal
uuid: "7c47bd901a35409e9687e6c2cd39191a"
format_stamp: "Formatted at 2026-09-27 10:48:16 on dist-test-slave-dzhw"
server_key: "a890daca5179e7abb1f2367ce4f6f734"
server_key_iv: "eb9530b7d4599cca015f6c056fe08752"
server_key_version: "encryptionkey@0"
I20260927 10:48:16.678627 22801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:16.686373 22719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } attempt: 3
I20260927 10:48:16.687548 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.687157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32885 (local address 127.0.0.1:57366)
0927 10:48:16.687345 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.687427 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 10:48:16.688383 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.688174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32885 (local address 127.0.0.1:57368)
0927 10:48:16.688278 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.688329 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 10:48:16.688540 22719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32885: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32885: connect: Connection refused (error 111)
I20260927 10:48:16.702368 22794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } attempt: 3
I20260927 10:48:16.703429 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.703079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32885 (local address 127.0.0.1:57380)
0927 10:48:16.703252 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.703331 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 10:48:16.704425 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.704108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32885 (local address 127.0.0.1:57388)
0927 10:48:16.704302 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.704361 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260927 10:48:16.704692 22794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32885: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32885: connect: Connection refused (error 111)
I20260927 10:48:16.705014 22801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:16.708969 22801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:16.709250 22801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:16.718168 22801 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:32885
I20260927 10:48:16.718668 22868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:32885 every 8 connection(s)
I20260927 10:48:16.719483 22801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-2/data/info.pb
I20260927 10:48:16.723153 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 22801
I20260927 10:48:16.723284 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505812134190-392-0/minicluster-data/master-2/wal/instance
I20260927 10:48:16.723464 22869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:16.723695 392 external_mini_cluster.cc:1428] Setting key 82baf0e07b53cd819bd81c56cedcdd1e
I20260927 10:48:16.730528 22869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:48:16 dist-test-slave-dzhw krb5kdc[22638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 10:48:16 dist-test-slave-dzhw krb5kdc[22638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:48:16.743197 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.725549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57398 (local address 127.0.98.60:32885)
0927 10:48:16.726154 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:16.726177 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:16.726230 (+ 53us) server_negotiation.cc:407] Connection header received
0927 10:48:16.727361 (+ 1131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:16.727393 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:16.728047 (+ 654us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:16.728552 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:16.729313 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.731120 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.732141 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.732858 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.733074 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:16.736467 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:16.736473 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:16.736486 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:16.736519 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:16.738779 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.739341 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.739348 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.739356 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.739514 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.739828 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.739835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.739837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.740562 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:16.740856 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:16.741107 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 10:48:16.741551 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":173,"threads_started":1}
I20260927 10:48:16.743563 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.733123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59116 (local address 127.0.98.62:33301)
0927 10:48:16.733297 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:16.733302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:16.733733 (+ 431us) server_negotiation.cc:407] Connection header received
0927 10:48:16.733968 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:16.733975 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:16.734055 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:16.734167 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:16.734963 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.735682 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.736687 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.737077 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.737208 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:16.739596 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:16.739603 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:16.739608 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:16.739642 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:16.741423 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.741968 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.741972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.741974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.742109 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.742398 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.742402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.742403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.742681 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:16.742795 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:16.743233 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 10:48:16.743388 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 10:48:16.743608 22873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.733032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33301 (local address 127.0.0.1:59116)
0927 10:48:16.733575 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.733632 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:16.733839 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:16.734220 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:16.734225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:16.734675 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:16.734820 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:16.734825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.735819 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.735825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:16.736558 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:16.736568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.737198 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.737201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:16.737294 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:16.737754 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:16.737766 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:16.739443 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:16.741553 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:16.741561 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:16.741567 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:16.741797 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:16.742216 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:16.742219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:16.742220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:16.742303 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:16.742845 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:16.742851 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:16.742978 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 10:48:16.743129 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 10:48:16.743381 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":195,"threads_started":1}
I20260927 10:48:16.744983 22869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33257 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:48:16 dist-test-slave-dzhw krb5kdc[22638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 10:48:16 dist-test-slave-dzhw krb5kdc[22638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:48:16.754667 22873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.745677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33257 (local address 127.0.0.1:49752)
0927 10:48:16.745814 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.745834 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:16.745924 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:16.746444 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:16.746448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:16.746646 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:16.746760 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:16.746765 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.747576 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.747586 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:16.748143 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:16.748151 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.748625 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.748628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:16.748726 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:16.749297 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:16.749317 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:16.750756 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:16.752874 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:16.752877 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:16.752879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:16.753088 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:16.753419 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:16.753425 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:16.753427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:16.753526 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:16.754029 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:16.754032 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:16.754153 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 10:48:16.754267 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 10:48:16.754400 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 10:48:16.754731 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.746028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49752 (local address 127.0.98.61:33257)
0927 10:48:16.746172 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:16.746179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:16.746196 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:48:16.746253 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:16.746256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:16.746303 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:16.746402 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:16.746882 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.747442 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.748264 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.748504 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.748623 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:16.750907 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:16.750911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:16.750913 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:16.750941 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:16.752760 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.753189 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.753194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.753196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.753296 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.753626 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.753628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.753629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.753899 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:16.754011 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:16.754346 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 10:48:16.754537 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:48:16.755641 22869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:48:16.755787 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.746129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59124 (local address 127.0.98.62:33301)
0927 10:48:16.746293 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:16.746296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:16.746310 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:48:16.746392 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:16.746395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:16.746466 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:16.746542 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:16.747232 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.747745 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.748536 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.748780 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.748884 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:16.751653 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:16.751657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:16.751661 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:16.751693 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:16.753335 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.753826 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.753829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.753831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.753913 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.754198 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.754200 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.754201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.754506 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:16.755364 (+ 858us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:16.755446 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 10:48:16.755616 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 10:48:16 dist-test-slave-dzhw krb5kdc[22638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 10:48:16 dist-test-slave-dzhw krb5kdc[22638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 10:48:16.765290 22873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.756450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32885 (local address 127.0.0.1:57408)
0927 10:48:16.756580 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.756597 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:16.756686 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:16.757019 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:16.757025 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:16.757228 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:16.757344 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:16.757350 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.758150 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.758156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:16.758786 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:16.758795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.759295 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.759298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:16.759444 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:16.760002 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:16.760018 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:16.761621 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:16.763599 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:16.763602 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:16.763603 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:16.763794 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:16.764093 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:16.764095 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:16.764097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:16.764180 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:16.764600 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:16.764603 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:16.764702 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 10:48:16.764815 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 10:48:16.765110 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":175}
I20260927 10:48:16.765290 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.756636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57408 (local address 127.0.98.60:32885)
0927 10:48:16.756764 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:16.756769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:16.756783 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:48:16.756818 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:16.756823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:16.756874 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:16.756953 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:16.757478 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.758040 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.758932 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.759185 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.759297 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:16.761827 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:16.761831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:16.761836 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:16.761867 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:16.763446 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.763897 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.763904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.763906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.763989 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.764276 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.764279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.764281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.764492 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:16.764608 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:16.764860 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 10:48:16.765051 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":50}
I20260927 10:48:16.766737 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.757244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49768 (local address 127.0.98.61:33257)
0927 10:48:16.757375 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:16.757379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:16.757392 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:48:16.757514 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:16.757518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:16.757563 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:16.757656 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:16.758301 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.758910 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.759735 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.760043 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.760156 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:16.763166 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:16.763170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:16.763172 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:16.763206 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:16.764735 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.765233 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.765237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.765239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.765329 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.765694 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.765697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.765699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.765939 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:16.766195 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:16.766342 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 10:48:16.766544 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:48:16.770805 22869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c47bd901a35409e9687e6c2cd39191a: Bootstrap starting.
I20260927 10:48:16.771236 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:48:16.772876 22693 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "49a02de860cc4112afdb4de6f3e149fd"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59124
I20260927 10:48:16.774389 22869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c47bd901a35409e9687e6c2cd39191a: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:16.775454 22869 log.cc:824] T 00000000000000000000000000000000 P 7c47bd901a35409e9687e6c2cd39191a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:16.778334 22869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c47bd901a35409e9687e6c2cd39191a: No bootstrap required, opened a new log
I20260927 10:48:16.784045 22869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c47bd901a35409e9687e6c2cd39191a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a02de860cc4112afdb4de6f3e149fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "8e0540b0e4d645159229541979245acc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33257 } } peers { permanent_uuid: "7c47bd901a35409e9687e6c2cd39191a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } }
I20260927 10:48:16.784490 22869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c47bd901a35409e9687e6c2cd39191a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c47bd901a35409e9687e6c2cd39191a, State: Initialized, Role: FOLLOWER
I20260927 10:48:16.784964 22869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c47bd901a35409e9687e6c2cd39191a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a02de860cc4112afdb4de6f3e149fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "8e0540b0e4d645159229541979245acc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33257 } } peers { permanent_uuid: "7c47bd901a35409e9687e6c2cd39191a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } }
I20260927 10:48:16.785688 22875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c47bd901a35409e9687e6c2cd39191a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a02de860cc4112afdb4de6f3e149fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "8e0540b0e4d645159229541979245acc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33257 } } peers { permanent_uuid: "7c47bd901a35409e9687e6c2cd39191a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } } }
I20260927 10:48:16.785885 22875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c47bd901a35409e9687e6c2cd39191a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:48:16.787074 22869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c47bd901a35409e9687e6c2cd39191a [sys.catalog]: configured and running, proceeding with master startup.
W20260927 10:48:16.793460 22886 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7c47bd901a35409e9687e6c2cd39191a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:48:16.793578 22886 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:48:16.792758 22801 master_runner.cc:186] Error getting master registration for 127.0.98.62:33301: OK, instance_id { permanent_uuid: "49a02de860cc4112afdb4de6f3e149fd" instance_seqno: 1790506096202365 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 10:48:16.797843 22801 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:48:16.847932 22794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } attempt: 4
I20260927 10:48:16.852751 22719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } attempt: 4
Sep 27 10:48:16 dist-test-slave-dzhw krb5kdc[22638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 10:48:16 dist-test-slave-dzhw krb5kdc[22638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 10:48:16.859532 22872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.848884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57412 (local address 127.0.98.60:32885)
0927 10:48:16.849033 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:16.849041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:16.849093 (+ 52us) server_negotiation.cc:407] Connection header received
0927 10:48:16.849302 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:16.849305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:16.849370 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:16.849458 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:16.849993 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.850636 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.851529 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.851882 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.852004 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:16.854995 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:16.854999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:16.855002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:16.855036 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:16.857569 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.858099 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.858103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.858105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.858194 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.858510 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.858515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.858520 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.858811 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:16.858928 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:16.859216 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 10:48:16.859340 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:48:16.859535 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.848791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32885 (local address 127.0.0.1:57412)
0927 10:48:16.848988 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.849026 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:16.849179 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:16.849506 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:16.849510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:16.849751 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:16.849863 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:16.849869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.850749 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.850754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:16.851402 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:16.851416 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.852029 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.852032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:16.852101 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:16.852697 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:16.852731 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:16.854865 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:16.857677 (+ 2812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:16.857681 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:16.857684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:16.857985 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:16.858301 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:16.858304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:16.858306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:16.858402 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:16.858917 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:16.858921 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:16.859041 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 10:48:16.859152 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 10:48:16.859329 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 10:48:16.863613 22794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e0540b0e4d645159229541979245acc: Bootstrap starting.
I20260927 10:48:16.863982 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.853385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32885 (local address 127.0.0.1:57416)
0927 10:48:16.854273 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:16.854287 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:16.854409 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:16.854731 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:16.854735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:16.854966 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:16.855104 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:16.855112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.856156 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.856160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:16.856728 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:16.856736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.857428 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.857434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:16.857544 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:16.858006 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:16.858020 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:16.859809 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:16.862108 (+ 2299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:16.862112 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:16.862114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:16.862379 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:16.862770 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:16.862786 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:16.862788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:16.862886 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:16.863356 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:16.863372 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:16.863480 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 10:48:16.863562 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 10:48:16.863776 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786}
I20260927 10:48:16.865273 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.853500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57416 (local address 127.0.98.60:32885)
0927 10:48:16.853976 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:16.853985 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:16.854353 (+ 368us) server_negotiation.cc:407] Connection header received
0927 10:48:16.854524 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:16.854529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:16.854594 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:16.854690 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:16.855227 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.856038 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.856851 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.857273 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.857781 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:16.860176 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:16.860180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:16.860183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:16.860207 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:16.861949 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.862567 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.862571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.862573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.862654 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.863015 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.863018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.863020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.863234 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:16.864437 (+ 1203us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:16.864527 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 10:48:16.864649 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":194,"threads_started":1}
I20260927 10:48:16.867094 22794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e0540b0e4d645159229541979245acc: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:16.868202 22794 log.cc:824] T 00000000000000000000000000000000 P 8e0540b0e4d645159229541979245acc: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:16.869313 22719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd: Bootstrap starting.
I20260927 10:48:16.870965 22794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e0540b0e4d645159229541979245acc: No bootstrap required, opened a new log
I20260927 10:48:16.872503 22719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:16.873503 22719 log.cc:824] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:16.875535 22794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e0540b0e4d645159229541979245acc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a02de860cc4112afdb4de6f3e149fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "8e0540b0e4d645159229541979245acc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33257 } } peers { permanent_uuid: "7c47bd901a35409e9687e6c2cd39191a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } }
I20260927 10:48:16.875751 22794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e0540b0e4d645159229541979245acc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0540b0e4d645159229541979245acc, State: Initialized, Role: FOLLOWER
I20260927 10:48:16.875929 22719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd: No bootstrap required, opened a new log
I20260927 10:48:16.876068 22794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e0540b0e4d645159229541979245acc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a02de860cc4112afdb4de6f3e149fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "8e0540b0e4d645159229541979245acc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33257 } } peers { permanent_uuid: "7c47bd901a35409e9687e6c2cd39191a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } }
I20260927 10:48:16.876543 22891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e0540b0e4d645159229541979245acc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a02de860cc4112afdb4de6f3e149fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "8e0540b0e4d645159229541979245acc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33257 } } peers { permanent_uuid: "7c47bd901a35409e9687e6c2cd39191a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } } }
I20260927 10:48:16.876689 22891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e0540b0e4d645159229541979245acc [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:48:16.877467 22794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e0540b0e4d645159229541979245acc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:48:16.879215 22719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a02de860cc4112afdb4de6f3e149fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "8e0540b0e4d645159229541979245acc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33257 } } peers { permanent_uuid: "7c47bd901a35409e9687e6c2cd39191a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } }
I20260927 10:48:16.879446 22719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49a02de860cc4112afdb4de6f3e149fd, State: Initialized, Role: FOLLOWER
I20260927 10:48:16.880332 22719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a02de860cc4112afdb4de6f3e149fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "8e0540b0e4d645159229541979245acc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33257 } } peers { permanent_uuid: "7c47bd901a35409e9687e6c2cd39191a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } }
I20260927 10:48:16.882323 22719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd [sys.catalog]: configured and running, proceeding with master startup.
W20260927 10:48:16.883765 22902 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e0540b0e4d645159229541979245acc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:48:16.883850 22902 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 10:48:16.888279 22915 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:48:16.888509 22915 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:48:16.891245 22693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 10:48:16.891415 22693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:16.893108 22693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a02de860cc4112afdb4de6f3e149fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "8e0540b0e4d645159229541979245acc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33257 } } peers { permanent_uuid: "7c47bd901a35409e9687e6c2cd39191a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } }
I20260927 10:48:16.893577 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a02de860cc4112afdb4de6f3e149fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "8e0540b0e4d645159229541979245acc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33257 } } peers { permanent_uuid: "7c47bd901a35409e9687e6c2cd39191a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } } }
I20260927 10:48:16.895352 22768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49a02de860cc4112afdb4de6f3e149fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8e0540b0e4d645159229541979245acc"
I20260927 10:48:16.895645 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:48:16.895658 22768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e0540b0e4d645159229541979245acc [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:16.897274 22768 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8e0540b0e4d645159229541979245acc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49a02de860cc4112afdb4de6f3e149fd in term 1.
I20260927 10:48:16.897799 22667 leader_election.cc:302] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49a02de860cc4112afdb4de6f3e149fd, 8e0540b0e4d645159229541979245acc; no voters:
I20260927 10:48:16.899870 22651 master_runner.cc:263] No leader master found from master 49a02de860cc4112afdb4de6f3e149fd
I20260927 10:48:16.899950 22651 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:48:16.881557 22726 master_runner.cc:186] Error getting master registration for 127.0.98.62:33301: OK, instance_id { permanent_uuid: "49a02de860cc4112afdb4de6f3e149fd" instance_seqno: 1790506096202365 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 10:48:16.901306 22726 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:48:16.902025 22843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49a02de860cc4112afdb4de6f3e149fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7c47bd901a35409e9687e6c2cd39191a"
I20260927 10:48:16.902326 22843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c47bd901a35409e9687e6c2cd39191a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:16.904048 22843 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7c47bd901a35409e9687e6c2cd39191a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49a02de860cc4112afdb4de6f3e149fd in term 1.
I20260927 10:48:16.904951 22693 leader_election.cc:288] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd [CANDIDATE]: Term 1 election: Requested vote from peers 8e0540b0e4d645159229541979245acc (127.0.98.61:33257), 7c47bd901a35409e9687e6c2cd39191a (127.0.98.60:32885)
I20260927 10:48:16.907708 22903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:16.908066 22903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd [term 1 LEADER]: Becoming Leader. State: Replica: 49a02de860cc4112afdb4de6f3e149fd, State: Running, Role: LEADER
I20260927 10:48:16.908558 22903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a02de860cc4112afdb4de6f3e149fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "8e0540b0e4d645159229541979245acc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33257 } } peers { permanent_uuid: "7c47bd901a35409e9687e6c2cd39191a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } }
I20260927 10:48:16.911343 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49a02de860cc4112afdb4de6f3e149fd. Latest consensus state: current_term: 1 leader_uuid: "49a02de860cc4112afdb4de6f3e149fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a02de860cc4112afdb4de6f3e149fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "8e0540b0e4d645159229541979245acc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33257 } } peers { permanent_uuid: "7c47bd901a35409e9687e6c2cd39191a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } } }
I20260927 10:48:16.911525 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:16.912029 22923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:48:16.913269 22923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:48:16.940164 22843 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7c47bd901a35409e9687e6c2cd39191a [term 1 FOLLOWER]: Refusing update from remote peer 49a02de860cc4112afdb4de6f3e149fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:48:16.941200 22921 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c47bd901a35409e9687e6c2cd39191a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:48:16.942794 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.914920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59128 (local address 127.0.98.62:33301)
0927 10:48:16.915078 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:16.915083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:16.919076 (+ 3993us) server_negotiation.cc:407] Connection header received
0927 10:48:16.919318 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:16.919323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:16.919408 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:16.919469 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:16.930560 (+ 11091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.931438 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.932732 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.933024 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.933107 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:16.935042 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:16.935070 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:16.935073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:16.935111 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:16.937011 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.941135 (+ 4124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.941141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.941143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.941259 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.941626 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.941630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.941632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.941910 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:16.942041 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:16.942459 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 10:48:16.942627 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 10:48:16.946648 22747 tablet.cc:2369] T 00000000000000000000000000000000 P 8e0540b0e4d645159229541979245acc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:48:16.948182 22672 tablet.cc:2369] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:48:16.950383 22875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c47bd901a35409e9687e6c2cd39191a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49a02de860cc4112afdb4de6f3e149fd. Latest consensus state: current_term: 1 leader_uuid: "49a02de860cc4112afdb4de6f3e149fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a02de860cc4112afdb4de6f3e149fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "8e0540b0e4d645159229541979245acc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33257 } } peers { permanent_uuid: "7c47bd901a35409e9687e6c2cd39191a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } } }
I20260927 10:48:16.950585 22875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c47bd901a35409e9687e6c2cd39191a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:48:16.954782 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.917758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49784 (local address 127.0.98.61:33257)
0927 10:48:16.917904 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:16.917909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:16.920606 (+ 2697us) server_negotiation.cc:407] Connection header received
0927 10:48:16.920793 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:16.920797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:16.920862 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:16.920919 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:16.923412 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.924285 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.927780 (+ 3495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.928192 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.928282 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:16.937029 (+ 8747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:16.937036 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:16.937041 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:16.937088 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:16.940779 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.950777 (+ 9998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.950783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.950785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.950921 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.951741 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.951746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.951748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.952045 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:16.952192 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:16.954397 (+ 2205us) server_negotiation.cc:299] Negotiation successful
0927 10:48:16.954594 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:48:16.954743 22928 mvcc.cc:205] Tried to move back new op lower bound from 7333912973047947264 to 7333912972942626816. Current Snapshot: MvccSnapshot[applied={T|T < 7333912973047947264}]
I20260927 10:48:16.955046 22875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c47bd901a35409e9687e6c2cd39191a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "49a02de860cc4112afdb4de6f3e149fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a02de860cc4112afdb4de6f3e149fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "8e0540b0e4d645159229541979245acc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33257 } } peers { permanent_uuid: "7c47bd901a35409e9687e6c2cd39191a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } } }
I20260927 10:48:16.955186 22875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c47bd901a35409e9687e6c2cd39191a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:48:16.957195 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:16.911003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57418 (local address 127.0.98.60:32885)
0927 10:48:16.911193 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:16.911201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:16.922069 (+ 10868us) server_negotiation.cc:407] Connection header received
0927 10:48:16.922258 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:16.922262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:16.922328 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:16.922388 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:16.925550 (+ 3162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.926451 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.929231 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:16.929596 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:16.929678 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:16.937507 (+ 7829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:16.937512 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:16.937515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:16.937560 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:16.939419 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.951329 (+ 11910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.951334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.951336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.951438 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:16.952532 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:16.952536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:16.952538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:16.952778 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:16.952887 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:16.956836 (+ 3949us) server_negotiation.cc:299] Negotiation successful
0927 10:48:16.957023 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:48:16.960132 22768 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e0540b0e4d645159229541979245acc [term 1 FOLLOWER]: Refusing update from remote peer 49a02de860cc4112afdb4de6f3e149fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:48:16.960848 22921 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0540b0e4d645159229541979245acc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:48:16.966208 22891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e0540b0e4d645159229541979245acc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49a02de860cc4112afdb4de6f3e149fd. Latest consensus state: current_term: 1 leader_uuid: "49a02de860cc4112afdb4de6f3e149fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a02de860cc4112afdb4de6f3e149fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "8e0540b0e4d645159229541979245acc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33257 } } peers { permanent_uuid: "7c47bd901a35409e9687e6c2cd39191a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } } }
I20260927 10:48:16.966374 22891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e0540b0e4d645159229541979245acc [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:48:16.966491 22891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e0540b0e4d645159229541979245acc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "49a02de860cc4112afdb4de6f3e149fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a02de860cc4112afdb4de6f3e149fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "8e0540b0e4d645159229541979245acc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33257 } } peers { permanent_uuid: "7c47bd901a35409e9687e6c2cd39191a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } } }
I20260927 10:48:16.966572 22891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e0540b0e4d645159229541979245acc [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:48:16.969843 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "49a02de860cc4112afdb4de6f3e149fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a02de860cc4112afdb4de6f3e149fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "8e0540b0e4d645159229541979245acc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33257 } } peers { permanent_uuid: "7c47bd901a35409e9687e6c2cd39191a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } } }
I20260927 10:48:16.970026 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:16.970996 22927 mvcc.cc:205] Tried to move back new op lower bound from 7333912973047947264 to 7333912972942626816. Current Snapshot: MvccSnapshot[applied={T|T < 7333912973047947264}]
I20260927 10:48:16.975616 22932 mvcc.cc:205] Tried to move back new op lower bound from 7333912973047947264 to 7333912972942626816. Current Snapshot: MvccSnapshot[applied={T|T < 7333912973047947264}]
I20260927 10:48:16.982062 22923 catalog_manager.cc:1347] Generated new cluster ID: 673b4b1b2da04814822d1b195238292a
I20260927 10:48:16.982178 22923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:48:17.019136 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "49a02de860cc4112afdb4de6f3e149fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49a02de860cc4112afdb4de6f3e149fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33301 } } peers { permanent_uuid: "8e0540b0e4d645159229541979245acc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33257 } } peers { permanent_uuid: "7c47bd901a35409e9687e6c2cd39191a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32885 } } }
I20260927 10:48:17.019335 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:17.024473 22923 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:48:17.026007 22923 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:48:17.036454 22923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49a02de860cc4112afdb4de6f3e149fd: Generated new TSK 0
I20260927 10:48:17.037663 22923 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:48:17.076349 22888 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:17.074311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57420 (local address 127.0.98.60:32885)
0927 10:48:17.074519 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:17.074524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:17.074808 (+ 284us) server_negotiation.cc:407] Connection header received
0927 10:48:17.075286 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:17.075289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:17.075348 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:17.075414 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:17.076160 (+ 746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57420: BlockingRecv error: recv got EOF from 127.0.0.1:57420 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 10:48:17.076675 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:17.075059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49786 (local address 127.0.98.61:33257)
0927 10:48:17.075210 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:17.075215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:17.075491 (+ 276us) server_negotiation.cc:407] Connection header received
0927 10:48:17.075711 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:17.075716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:17.075786 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:17.075867 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:17.076515 (+ 648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49786: BlockingRecv error: recv got EOF from 127.0.0.1:49786 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 10:48:17.077133 22723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:17.074859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59142 (local address 127.0.98.62:33301)
0927 10:48:17.075025 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:17.075032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:17.075051 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:48:17.075170 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:17.075174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:17.075249 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:17.075380 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:17.076995 (+ 1615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59142: BlockingRecv error: recv got EOF from 127.0.0.1:59142 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 10:48:17.084431 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:17.082500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49802 (local address 127.0.98.61:33257)
0927 10:48:17.082644 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:17.082649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:17.082927 (+ 278us) server_negotiation.cc:407] Connection header received
0927 10:48:17.083234 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:17.083238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:17.083340 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:17.083452 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:17.084245 (+ 793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49802: BlockingRecv error: recv got EOF from 127.0.0.1:49802 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 10:48:17.089668 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:17.088534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49812 (local address 127.0.98.61:33257)
0927 10:48:17.088681 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:17.088685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:17.088701 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:48:17.088882 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:17.088885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:17.088946 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:17.089038 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:17.089534 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49812: BlockingRecv error: recv got EOF from 127.0.0.1:49812 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 10:48:17.095552 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:17.094223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49822 (local address 127.0.98.61:33257)
0927 10:48:17.094340 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:17.094345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:17.094584 (+ 239us) server_negotiation.cc:407] Connection header received
0927 10:48:17.094707 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:17.094710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:17.094767 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:17.094852 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:17.095414 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49822: BlockingRecv error: recv got EOF from 127.0.0.1:49822 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 10:48:17.104450 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:17.102992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49826 (local address 127.0.98.61:33257)
0927 10:48:17.103118 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:17.103123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:17.103373 (+ 250us) server_negotiation.cc:407] Connection header received
0927 10:48:17.103526 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:17.103530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:17.103587 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:17.103677 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:17.104302 (+ 625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49826: BlockingRecv error: recv got EOF from 127.0.0.1:49826 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 10:48:17.116058 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:17.114729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49838 (local address 127.0.98.61:33257)
0927 10:48:17.114911 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:17.114918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:17.115064 (+ 146us) server_negotiation.cc:407] Connection header received
0927 10:48:17.115199 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:17.115201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:17.115252 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:17.115324 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:17.115891 (+ 567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49838: BlockingRecv error: recv got EOF from 127.0.0.1:49838 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 10:48:17.135888 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:17.134549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49848 (local address 127.0.98.61:33257)
0927 10:48:17.134756 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:17.134761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:17.134865 (+ 104us) server_negotiation.cc:407] Connection header received
0927 10:48:17.135027 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:17.135029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:17.135089 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:17.135168 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:17.135709 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49848: BlockingRecv error: recv got EOF from 127.0.0.1:49848 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260927 10:48:17.172025 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:17.170582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49860 (local address 127.0.98.61:33257)
0927 10:48:17.170809 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:17.170814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:17.170896 (+ 82us) server_negotiation.cc:407] Connection header received
0927 10:48:17.171045 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:17.171048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:17.171112 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:17.171200 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:17.171848 (+ 648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49860: BlockingRecv error: recv got EOF from 127.0.0.1:49860 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260927 10:48:17.240139 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:17.238580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49870 (local address 127.0.98.61:33257)
0927 10:48:17.238804 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:17.238810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:17.238986 (+ 176us) server_negotiation.cc:407] Connection header received
0927 10:48:17.239202 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:17.239205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:17.239272 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:17.239417 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:17.239959 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49870: BlockingRecv error: recv got EOF from 127.0.0.1:49870 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260927 10:48:17.372381 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:17.370871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49874 (local address 127.0.98.61:33257)
0927 10:48:17.371024 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:17.371032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:17.371238 (+ 206us) server_negotiation.cc:407] Connection header received
0927 10:48:17.371453 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:17.371455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:17.371563 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:17.371664 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:17.372216 (+ 552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49874: BlockingRecv error: recv got EOF from 127.0.0.1:49874 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 10:48:17.632442 22796 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:17.631082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49884 (local address 127.0.98.61:33257)
0927 10:48:17.631299 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:17.631304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:17.631327 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:48:17.631497 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:17.631500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:17.631566 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:17.631657 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:17.632289 (+ 632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49884: BlockingRecv error: recv got EOF from 127.0.0.1:49884 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 10:48:17.795495 22886 catalog_manager.cc:1259] Loaded cluster ID: 673b4b1b2da04814822d1b195238292a
I20260927 10:48:17.795612 22886 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7c47bd901a35409e9687e6c2cd39191a: loading cluster ID for follower catalog manager: success
I20260927 10:48:17.797797 22886 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7c47bd901a35409e9687e6c2cd39191a: acquiring CA information for follower catalog manager: success
I20260927 10:48:17.799367 22886 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7c47bd901a35409e9687e6c2cd39191a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:48:17.885706 22902 catalog_manager.cc:1259] Loaded cluster ID: 673b4b1b2da04814822d1b195238292a
I20260927 10:48:17.885831 22902 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8e0540b0e4d645159229541979245acc: loading cluster ID for follower catalog manager: success
I20260927 10:48:17.888188 22902 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8e0540b0e4d645159229541979245acc: acquiring CA information for follower catalog manager: success
I20260927 10:48:17.889727 22902 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8e0540b0e4d645159229541979245acc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:48:18.152236 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:18.147412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49892 (local address 127.0.98.61:33257)
0927 10:48:18.147735 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:18.147741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:18.147886 (+ 145us) server_negotiation.cc:407] Connection header received
0927 10:48:18.148128 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:18.148131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:18.148321 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:18.148471 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:48:18.148776 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:18.149390 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:18.150330 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:18.150658 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:18.150778 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:18.151159 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:48:18.151762 (+ 603us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:48:18.151913 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:18.151950 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 10:48:18.152034 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":89,"threads_started":1}
I20260927 10:48:18.160827 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:18.156406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57432 (local address 127.0.98.60:32885)
0927 10:48:18.156748 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:18.156755 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:18.156774 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:48:18.156887 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:18.156890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:18.157059 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:18.157210 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:48:18.157455 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:18.158035 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:18.159024 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:18.159267 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:18.159382 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:18.159740 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:48:18.160357 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:48:18.160511 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:18.160551 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 10:48:18.160627 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":82,"threads_started":1}
I20260927 10:48:18.170913 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:18.165476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57440 (local address 127.0.98.60:32885)
0927 10:48:18.165833 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:18.165839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:18.166600 (+ 761us) server_negotiation.cc:407] Connection header received
0927 10:48:18.166759 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:18.166763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:18.166845 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:18.166946 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:48:18.167176 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:18.168067 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:18.169261 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:18.169606 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:18.169718 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:18.170129 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:48:18.170292 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:48:18.170369 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:18.170634 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 10:48:18.170744 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222}
I20260927 10:48:18.170913 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:18.166010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49894 (local address 127.0.98.61:33257)
0927 10:48:18.166200 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:18.166206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:18.166954 (+ 748us) server_negotiation.cc:407] Connection header received
0927 10:48:18.167128 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:18.167131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:18.167200 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:18.167282 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:48:18.167552 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:18.168308 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:18.169433 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:18.169812 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:18.169945 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:18.170301 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:48:18.170400 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:48:18.170493 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:18.170684 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 10:48:18.170747 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 10:48:18.171689 23013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:18.165245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59152 (local address 127.0.98.62:33301)
0927 10:48:18.165702 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:18.165708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:18.166195 (+ 487us) server_negotiation.cc:407] Connection header received
0927 10:48:18.166407 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:18.166410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:18.166619 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:18.166774 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:48:18.167087 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:18.167860 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:18.169192 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:18.169525 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:18.169630 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:18.170141 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:48:18.170710 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:48:18.170806 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:18.171406 (+ 600us) server_negotiation.cc:299] Negotiation successful
0927 10:48:18.171493 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":72,"threads_started":1}
I20260927 10:48:18.176815 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 22651
I20260927 10:48:18.191425 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 22726
I20260927 10:48:18.200152 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 22801
2026-09-27T10:48:18Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3414 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:48:18 dist-test-slave-dzhw krb5kdc[23018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:48:18 dist-test-slave-dzhw krb5kdc[23018](info): set up 2 sockets
Sep 27 10:48:18 dist-test-slave-dzhw krb5kdc[23018](info): commencing operation
krb5kdc: starting...
W20260927 10:48:19.335788 392 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/krb5kdc/test-user.keytab.
Sep 27 10:48:19 dist-test-slave-dzhw krb5kdc[23018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:48:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:48:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 10:48:19.471139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36311
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36311,127.0.98.61:42327,127.0.98.60:37013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:19.612213 23031 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:48:19.612639 23031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:19.612666 23031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:19.612761 23031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:19.617390 23031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:19.617489 23031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:19.617511 23031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:19.617529 23031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:19.617547 23031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:19.623972 23031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36311,127.0.98.61:42327,127.0.98.60:37013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:48:19.625589 23031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:19.627031 23031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:19.637777 23036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:19.637758 23039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:19.637758 23037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:19.638227 23031 server_base.cc:1034] running on GCE node
I20260927 10:48:19.639056 23031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:19.640483 23031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:19.641695 23031 hybrid_clock.cc:648] HybridClock initialized: now 1790506099641662 us; error 48 us; skew 500 ppm
Sep 27 10:48:19 dist-test-slave-dzhw krb5kdc[23018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:19.645080 23031 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 10:48:19.646196 23031 webserver.cc:460] Webserver started at http://127.0.98.62:34525/ using document root <none> and password file <none>
I20260927 10:48:19.646962 23031 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:19.647063 23031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:19.647339 23031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:19.649930 23031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-0/data/instance:
uuid: "8f82289dec1b4b1f80e197a78bae2f07"
format_stamp: "Formatted at 2026-09-27 10:48:19 on dist-test-slave-dzhw"
server_key: "a340e96e720481311f744ca37c825afa"
server_key_iv: "816edd399e6ba8128bf3809920fa8559"
server_key_version: "encryptionkey@0"
I20260927 10:48:19.650695 23031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-0/wal/instance:
uuid: "8f82289dec1b4b1f80e197a78bae2f07"
format_stamp: "Formatted at 2026-09-27 10:48:19 on dist-test-slave-dzhw"
server_key: "a340e96e720481311f744ca37c825afa"
server_key_iv: "816edd399e6ba8128bf3809920fa8559"
server_key_version: "encryptionkey@0"
I20260927 10:48:19.655529 23031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 10:48:19.658630 23046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:19.660192 23031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:48:19.660346 23031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-0/wal
uuid: "8f82289dec1b4b1f80e197a78bae2f07"
format_stamp: "Formatted at 2026-09-27 10:48:19 on dist-test-slave-dzhw"
server_key: "a340e96e720481311f744ca37c825afa"
server_key_iv: "816edd399e6ba8128bf3809920fa8559"
server_key_version: "encryptionkey@0"
I20260927 10:48:19.660478 23031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:19.674552 23031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:19.678066 23031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:19.678308 23031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:19.687378 23031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36311
I20260927 10:48:19.687747 23098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36311 every 8 connection(s)
I20260927 10:48:19.688663 23031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-0/data/info.pb
I20260927 10:48:19.690002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 23031
I20260927 10:48:19.690145 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-0/wal/instance
I20260927 10:48:19.690552 392 external_mini_cluster.cc:1428] Setting key 896ac344582eab1b355e668956a870d0
I20260927 10:48:19.692613 23099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:19.697788 23099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36311 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 10:48:19 dist-test-slave-dzhw krb5kdc[23018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:48:19.717650 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:19.701368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48048 (local address 127.0.98.62:36311)
0927 10:48:19.702051 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:19.702082 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:19.702406 (+ 324us) server_negotiation.cc:407] Connection header received
0927 10:48:19.704106 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:19.704112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:19.704377 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:19.704608 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:19.705608 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:19.707069 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:19.708193 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:19.708920 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:19.709138 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:19.712054 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:19.712062 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:19.712081 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:19.712108 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:19.713877 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:19.714541 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:19.714569 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:19.714571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:19.714740 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:19.715121 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:19.715124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:19.715125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:19.715681 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:19.715798 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:19.716296 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 10:48:19.716515 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"spinlock_wait_cycles":91392,"thread_start_us":240,"threads_started":1}
I20260927 10:48:19.717651 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:19.701368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36311 (local address 127.0.0.1:48048)
0927 10:48:19.702168 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:19.702235 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:19.703706 (+ 1471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:19.704604 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:19.704610 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:19.705164 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:19.705451 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:19.705467 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:19.707219 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:19.707223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:19.708061 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:19.708073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:19.709156 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:19.709160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:19.709265 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:19.710042 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:19.710061 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:19.711858 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:19.714057 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:19.714071 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:19.714089 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:19.714350 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:19.714884 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:19.714886 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:19.714888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:19.715014 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:19.715801 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:19.715808 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:19.715953 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 10:48:19.716176 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 10:48:19.716610 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":93,"thread_start_us":201,"threads_started":1}
I20260927 10:48:19.720724 23099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 10:48:19.721895 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:19.721569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42327 (local address 127.0.0.1:50634)
0927 10:48:19.721721 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:19.721810 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 10:48:19.722184 23047 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42327: connect: Connection refused (error 111)
I20260927 10:48:19.722774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42327
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36311,127.0.98.61:42327,127.0.98.60:37013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:48:19.723268 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:19.723045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42327 (local address 127.0.0.1:50636)
0927 10:48:19.723156 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:19.723205 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 10:48:19.723493 23099 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42327: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42327: connect: Connection refused (error 111)
I20260927 10:48:19.772804 23099 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 } attempt: 1
I20260927 10:48:19.774165 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:19.773773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42327 (local address 127.0.0.1:50650)
0927 10:48:19.773974 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:19.774064 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 10:48:19.775058 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:19.774807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42327 (local address 127.0.0.1:50664)
0927 10:48:19.774925 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:19.774990 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 10:48:19.775199 23099 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42327: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42327: connect: Connection refused (error 111)
I20260927 10:48:19.843371 23099 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 } attempt: 2
I20260927 10:48:19.844461 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:19.844117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42327 (local address 127.0.0.1:50666)
0927 10:48:19.844287 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:19.844365 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 10:48:19.845310 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:19.845094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42327 (local address 127.0.0.1:50680)
0927 10:48:19.845215 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:19.845257 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 10:48:19.845424 23099 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42327: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42327: connect: Connection refused (error 111)
W20260927 10:48:19.857426 23106 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:48:19.857800 23106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:19.857825 23106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:19.857926 23106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:19.862309 23106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:19.862398 23106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:19.862489 23106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:19.862514 23106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:19.862556 23106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:19.868882 23106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36311,127.0.98.61:42327,127.0.98.60:37013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:48:19.870493 23106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:19.871831 23106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:19.882128 23114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:19.882139 23111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:19.882682 23106 server_base.cc:1034] running on GCE node
W20260927 10:48:19.882211 23112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:19.883466 23106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:19.884996 23106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:19.886206 23106 hybrid_clock.cc:648] HybridClock initialized: now 1790506099886173 us; error 56 us; skew 500 ppm
Sep 27 10:48:19 dist-test-slave-dzhw krb5kdc[23018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:19.889568 23106 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 10:48:19.890861 23106 webserver.cc:460] Webserver started at http://127.0.98.61:39321/ using document root <none> and password file <none>
I20260927 10:48:19.891610 23106 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:19.891712 23106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:19.892000 23106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:19.894814 23106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-1/data/instance:
uuid: "8fd4e777df6547d882318ec766f49745"
format_stamp: "Formatted at 2026-09-27 10:48:19 on dist-test-slave-dzhw"
server_key: "a8a542dc05b7cccf9205b2fe441ee465"
server_key_iv: "caf240be6071db0878494feabedb555e"
server_key_version: "encryptionkey@0"
I20260927 10:48:19.895519 23106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-1/wal/instance:
uuid: "8fd4e777df6547d882318ec766f49745"
format_stamp: "Formatted at 2026-09-27 10:48:19 on dist-test-slave-dzhw"
server_key: "a8a542dc05b7cccf9205b2fe441ee465"
server_key_iv: "caf240be6071db0878494feabedb555e"
server_key_version: "encryptionkey@0"
I20260927 10:48:19.900302 23106 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:48:19.903894 23121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:19.905658 23106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:48:19.905875 23106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-1/wal
uuid: "8fd4e777df6547d882318ec766f49745"
format_stamp: "Formatted at 2026-09-27 10:48:19 on dist-test-slave-dzhw"
server_key: "a8a542dc05b7cccf9205b2fe441ee465"
server_key_iv: "caf240be6071db0878494feabedb555e"
server_key_version: "encryptionkey@0"
I20260927 10:48:19.906055 23106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:19.927491 23106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:19.932422 23106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:19.932729 23106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:19.936657 23099 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 } attempt: 3
I20260927 10:48:19.937878 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:19.937510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42327 (local address 127.0.0.1:50686)
0927 10:48:19.937691 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:19.937771 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 10:48:19.938897 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:19.938515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42327 (local address 127.0.0.1:50692)
0927 10:48:19.938685 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:19.938769 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 10:48:19.939109 23099 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42327: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42327: connect: Connection refused (error 111)
I20260927 10:48:19.942498 23106 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42327
I20260927 10:48:19.942862 23173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42327 every 8 connection(s)
I20260927 10:48:19.943899 23106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-1/data/info.pb
I20260927 10:48:19.947767 23174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:19.951887 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 23106
I20260927 10:48:19.952049 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-1/wal/instance
I20260927 10:48:19.952418 392 external_mini_cluster.cc:1428] Setting key 828f68f62f9de6e5b82f98d46e34ce4f
I20260927 10:48:19.952961 23174 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36311 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 10:48:19 dist-test-slave-dzhw krb5kdc[23018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:48:19.970492 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:19.955907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48052 (local address 127.0.98.62:36311)
0927 10:48:19.956107 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:19.956113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:19.957185 (+ 1072us) server_negotiation.cc:407] Connection header received
0927 10:48:19.958393 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:19.958397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:19.958517 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:19.958621 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:19.960024 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:19.960613 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:19.961917 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:19.962175 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:19.962307 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:19.965726 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:19.965729 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:19.965732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:19.965763 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:19.967616 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:19.968261 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:19.968266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:19.968269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:19.968380 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:19.968792 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:19.968797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:19.968800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:19.969142 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:19.969275 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:19.970119 (+ 844us) server_negotiation.cc:299] Negotiation successful
0927 10:48:19.970269 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 10:48:19.972239 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:19.956381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36311 (local address 127.0.0.1:48052)
0927 10:48:19.957001 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:19.957062 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:19.958140 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:19.958831 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:19.958839 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:19.959265 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:19.959854 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:19.959877 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:19.960759 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:19.960763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:19.961783 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:19.961793 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:19.962351 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:19.962355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:19.962585 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:19.963254 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:19.963273 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:19.965515 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:19.967769 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:19.967777 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:19.967810 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:19.968137 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:19.968489 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:19.968494 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:19.968497 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:19.968681 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:19.969275 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:19.969285 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:19.969743 (+ 458us) client_negotiation.cc:769] Sending connection context
0927 10:48:19.970057 (+ 314us) client_negotiation.cc:240] Negotiation successful
0927 10:48:19.970635 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":174,"threads_started":1}
I20260927 10:48:19.974051 23174 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:48:19 dist-test-slave-dzhw krb5kdc[23018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 10:48:19.984376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
/tmp/dist-test-taskkqkM3n/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37013
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36311,127.0.98.61:42327,127.0.98.60:37013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:48:19.984731 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:19.974705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42327 (local address 127.0.0.1:50708)
0927 10:48:19.974890 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:19.974904 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:19.975004 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:19.975688 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:19.975691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:19.975903 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:19.975993 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:19.975999 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:19.977049 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:19.977053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:19.977608 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:19.977616 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:19.978282 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:19.978285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:19.978466 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:19.979055 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:19.979068 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:19.980630 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:19.982968 (+ 2338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:19.982974 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:19.982977 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:19.983178 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:19.983506 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:19.983512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:19.983516 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:19.983600 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:19.984128 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:19.984132 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:19.984220 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 10:48:19.984348 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 10:48:19.984510 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":20}
I20260927 10:48:19.984740 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:19.974883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50708 (local address 127.0.98.61:42327)
0927 10:48:19.975292 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:19.975303 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:19.975323 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:48:19.975359 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:19.975366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:19.975558 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:19.975722 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:19.976105 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:19.976929 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:19.977721 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:19.978162 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:19.978284 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:19.980781 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:19.980793 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:19.980823 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:19.980842 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:19.982849 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:19.983301 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:19.983311 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:19.983316 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:19.983428 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:19.983692 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:19.983695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:19.983697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:19.984019 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:19.984182 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:19.984444 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 10:48:19.984588 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":319,"thread_start_us":137,"threads_started":1}
I20260927 10:48:19.988699 23174 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:48:19.989809 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:19.989361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37013 (local address 127.0.0.1:57116)
0927 10:48:19.989554 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:19.989676 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 10:48:19.990123 23124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37013: connect: Connection refused (error 111)
I20260927 10:48:19.991405 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:19.991016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37013 (local address 127.0.0.1:57122)
0927 10:48:19.991178 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:19.991271 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 10:48:19.991601 23174 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37013: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37013: connect: Connection refused (error 111)
I20260927 10:48:20.040963 23174 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } attempt: 1
I20260927 10:48:20.042222 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.041827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37013 (local address 127.0.0.1:57130)
0927 10:48:20.042039 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:20.042116 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 10:48:20.043140 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.042901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37013 (local address 127.0.0.1:57142)
0927 10:48:20.043027 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:20.043077 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 10:48:20.043378 23174 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37013: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37013: connect: Connection refused (error 111)
I20260927 10:48:20.082331 23099 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 } attempt: 4
Sep 27 10:48:20 dist-test-slave-dzhw krb5kdc[23018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 10:48:20.093225 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.083260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42327 (local address 127.0.0.1:50714)
0927 10:48:20.083524 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:20.083543 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:20.083663 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:20.083977 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:20.083980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:20.084225 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:20.084349 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:20.084355 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.085213 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.085216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:20.086240 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:20.086249 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.087327 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.087331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:20.087463 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:20.088173 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:20.088186 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:20.089594 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:20.091555 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:20.091561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:20.091564 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:20.091778 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:20.092074 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:20.092078 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:20.092080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:20.092169 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:20.092609 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:20.092612 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:20.092730 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 10:48:20.092842 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 10:48:20.093025 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260927 10:48:20.093225 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.083378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50714 (local address 127.0.98.61:42327)
0927 10:48:20.083567 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:20.083574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:20.083615 (+ 41us) server_negotiation.cc:407] Connection header received
0927 10:48:20.083796 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:20.083805 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:20.083863 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:20.083958 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:20.084479 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.085084 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.086373 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.087189 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.087322 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:20.089751 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:20.089754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:20.089757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:20.089783 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:20.091411 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:20.091881 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:20.091886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:20.091888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:20.091971 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:20.092267 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:20.092271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:20.092273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:20.092510 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:20.092621 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:20.092896 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 10:48:20.093025 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 10:48:20.094583 23099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:48:20.095419 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.095159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37013 (local address 127.0.0.1:57156)
0927 10:48:20.095288 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:20.095351 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 10:48:20.096093 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.095890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37013 (local address 127.0.0.1:57164)
0927 10:48:20.095988 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:20.096035 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 10:48:20.096235 23099 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37013: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37013: connect: Connection refused (error 111)
I20260927 10:48:20.111569 23174 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } attempt: 2
I20260927 10:48:20.112739 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.112350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37013 (local address 127.0.0.1:57178)
0927 10:48:20.112569 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:20.112638 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 10:48:20.113658 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.113439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37013 (local address 127.0.0.1:57192)
0927 10:48:20.113550 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:20.113599 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 10:48:20.113895 23174 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37013: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37013: connect: Connection refused (error 111)
W20260927 10:48:20.130064 23180 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:48:20.130576 23180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:20.130636 23180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:20.130724 23180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:20.135234 23180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:20.135324 23180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:20.135347 23180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:20.135411 23180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:20.135452 23180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:20.141839 23180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36311,127.0.98.61:42327,127.0.98.60:37013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:05:23 UTC on ccycloud.quasar-frhndg.root.comops.site
build id 4675884
I20260927 10:48:20.143431 23180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:20.144796 23180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:20.154943 23187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:20.154960 23186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:20.154964 23189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:20.155408 23180 server_base.cc:1034] running on GCE node
I20260927 10:48:20.155388 23099 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } attempt: 1
I20260927 10:48:20.156160 23180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:20.156380 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.156005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37013 (local address 127.0.0.1:57198)
0927 10:48:20.156175 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:20.156237 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 10:48:20.157243 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.157010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37013 (local address 127.0.0.1:57200)
0927 10:48:20.157123 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:20.157175 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 10:48:20.157387 23099 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37013: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37013: connect: Connection refused (error 111)
I20260927 10:48:20.157773 23180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:20.158978 23180 hybrid_clock.cc:648] HybridClock initialized: now 1790506100158938 us; error 52 us; skew 500 ppm
Sep 27 10:48:20 dist-test-slave-dzhw krb5kdc[23018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:20.162235 23180 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 10:48:20.163406 23180 webserver.cc:460] Webserver started at http://127.0.98.60:43031/ using document root <none> and password file <none>
I20260927 10:48:20.164193 23180 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:20.164292 23180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:20.164575 23180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:20.167368 23180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-2/data/instance:
uuid: "7b343bb41ef2441aac0a4e614c8f1c46"
format_stamp: "Formatted at 2026-09-27 10:48:20 on dist-test-slave-dzhw"
server_key: "e408bbe5621c39a9f6a380719cb8d8e5"
server_key_iv: "855d167480425d340e4c89f2d2ef79f6"
server_key_version: "encryptionkey@0"
I20260927 10:48:20.168068 23180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-2/wal/instance:
uuid: "7b343bb41ef2441aac0a4e614c8f1c46"
format_stamp: "Formatted at 2026-09-27 10:48:20 on dist-test-slave-dzhw"
server_key: "e408bbe5621c39a9f6a380719cb8d8e5"
server_key_iv: "855d167480425d340e4c89f2d2ef79f6"
server_key_version: "encryptionkey@0"
I20260927 10:48:20.172796 23180 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:48:20.175997 23196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:20.177827 23180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 10:48:20.178009 23180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-2/wal
uuid: "7b343bb41ef2441aac0a4e614c8f1c46"
format_stamp: "Formatted at 2026-09-27 10:48:20 on dist-test-slave-dzhw"
server_key: "e408bbe5621c39a9f6a380719cb8d8e5"
server_key_iv: "855d167480425d340e4c89f2d2ef79f6"
server_key_version: "encryptionkey@0"
I20260927 10:48:20.178170 23180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:20.200713 23180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:20.204592 23180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:20.204859 23180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:20.205071 23174 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } attempt: 3
I20260927 10:48:20.206101 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.205708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37013 (local address 127.0.0.1:57206)
0927 10:48:20.205909 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:20.205992 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 10:48:20.207063 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.206720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37013 (local address 127.0.0.1:57220)
0927 10:48:20.206916 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:20.206971 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260927 10:48:20.207211 23174 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37013: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37013: connect: Connection refused (error 111)
I20260927 10:48:20.213873 23180 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37013
I20260927 10:48:20.214221 23248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37013 every 8 connection(s)
I20260927 10:48:20.215263 23180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-2/data/info.pb
I20260927 10:48:20.219010 23249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:20.223799 23249 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36311 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:48:20.224615 23099 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } attempt: 2
I20260927 10:48:20.224931 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu as pid 23180
I20260927 10:48:20.225153 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkqkM3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505812134190-392-0/minicluster-data/master-2/wal/instance
I20260927 10:48:20.225556 392 external_mini_cluster.cc:1428] Setting key ce2291cf48361383dc89aa5bb692f2cf
Sep 27 10:48:20 dist-test-slave-dzhw krb5kdc[23018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 10:48:20 dist-test-slave-dzhw krb5kdc[23018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 10:48:20 dist-test-slave-dzhw krb5kdc[23018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 10:48:20.241073 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.227265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48062 (local address 127.0.98.62:36311)
0927 10:48:20.227413 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:20.227419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:20.227763 (+ 344us) server_negotiation.cc:407] Connection header received
0927 10:48:20.228244 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:20.228247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:20.228296 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:20.228398 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:20.229850 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.230667 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.232061 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.232282 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.232434 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:20.236016 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:20.236020 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:20.236023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:20.236053 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:20.237768 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:20.238818 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:20.238825 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:20.238828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:20.238934 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:20.239373 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:20.239376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:20.239378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:20.239798 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:20.239905 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:20.240638 (+ 733us) server_negotiation.cc:299] Negotiation successful
0927 10:48:20.240806 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":62}
I20260927 10:48:20.241073 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.225348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37013 (local address 127.0.0.1:57228)
0927 10:48:20.225604 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:20.225622 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:20.225739 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:20.228521 (+ 2782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:20.228524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:20.228782 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:20.228917 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:20.228925 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.230881 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.230887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:20.231432 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:20.231439 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.232389 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.232392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:20.232623 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:20.233397 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:20.233412 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:20.235102 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:20.238925 (+ 3823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:20.238928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:20.238931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:20.239217 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:20.239561 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:20.239564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:20.239566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:20.239647 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:20.240517 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:20.240521 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:20.240633 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 10:48:20.240727 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 10:48:20.240872 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":138}
I20260927 10:48:20.242278 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.226684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57234 (local address 127.0.98.60:37013)
0927 10:48:20.226964 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:20.226972 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:20.226989 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:48:20.228016 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:20.228030 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:20.228304 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:20.229249 (+ 945us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:20.229525 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.230646 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.231851 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.232263 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.232889 (+ 626us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:20.236584 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:20.236587 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:20.236590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:20.236620 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:20.239394 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:20.239911 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:20.239914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:20.239915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:20.239992 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:20.240282 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:20.240285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:20.240286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:20.240492 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:20.241924 (+ 1432us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:20.241984 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 10:48:20.242077 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":218,"spinlock_wait_cycles":896,"thread_start_us":104,"threads_started":1}
I20260927 10:48:20.242285 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.226315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57228 (local address 127.0.98.60:37013)
0927 10:48:20.226910 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:20.226939 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:20.226987 (+ 48us) server_negotiation.cc:407] Connection header received
0927 10:48:20.228005 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:20.228029 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:20.228308 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:20.229079 (+ 771us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:20.229491 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.230702 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.231551 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.232263 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.232434 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:20.235280 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:20.235290 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:20.235310 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:20.235343 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:20.238817 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:20.239343 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:20.239352 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:20.239357 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:20.239443 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:20.239739 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:20.239744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:20.239746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:20.240383 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:20.240509 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:20.241200 (+ 691us) server_negotiation.cc:299] Negotiation successful
0927 10:48:20.241345 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":281,"server-negotiator.queue_time_us":416,"spinlock_wait_cycles":14464,"thread_start_us":167,"threads_started":1}
I20260927 10:48:20.242285 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.227191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36311 (local address 127.0.0.1:48062)
0927 10:48:20.227605 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:20.227660 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:20.228018 (+ 358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:20.228457 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:20.228468 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:20.228946 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:20.229701 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:20.229715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.230796 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.230799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:20.231949 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:20.231957 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.232505 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.232509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:20.232623 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:20.233396 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:20.233469 (+ 73us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:20.235853 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:20.238043 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:20.238054 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:20.238069 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:20.238675 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:20.239045 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:20.239050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:20.239052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:20.239254 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:20.239968 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:20.239974 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:20.240374 (+ 400us) client_negotiation.cc:769] Sending connection context
0927 10:48:20.240592 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 10:48:20.241003 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":25,"spinlock_wait_cycles":4224,"thread_start_us":109,"threads_started":1}
I20260927 10:48:20.244633 23249 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:48:20.248106 23099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07: Bootstrap starting.
Sep 27 10:48:20 dist-test-slave-dzhw krb5kdc[23018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 10:48:20 dist-test-slave-dzhw krb5kdc[23018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 10:48:20.251909 23099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:20.253069 23099 log.cc:824] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:20.255399 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.245590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50726 (local address 127.0.98.61:42327)
0927 10:48:20.245718 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:20.245724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:20.246657 (+ 933us) server_negotiation.cc:407] Connection header received
0927 10:48:20.246774 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:20.246779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:20.246839 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:20.246930 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:20.247354 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.248095 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.249062 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.249376 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.249490 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:20.251650 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:20.251657 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:20.251664 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:20.251702 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:20.253376 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:20.253894 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:20.253898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:20.253900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:20.253996 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:20.254309 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:20.254312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:20.254314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:20.254630 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:20.254708 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:20.255073 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 10:48:20.255220 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 10:48:20.255970 23099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07: No bootstrap required, opened a new log
I20260927 10:48:20.255998 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.245213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42327 (local address 127.0.0.1:50726)
0927 10:48:20.246570 (+ 1357us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:20.246582 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:20.246681 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:20.246940 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:20.246943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:20.247126 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:20.247238 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:20.247247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.248216 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.248220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:20.248932 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:20.248944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.249504 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.249507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:20.249582 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:20.250050 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:20.250061 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:20.251495 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:20.253522 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:20.253534 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:20.253539 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:20.253773 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:20.254092 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:20.254098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:20.254100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:20.254196 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:20.254781 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:20.254783 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:20.254886 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 10:48:20.255670 (+ 784us) client_negotiation.cc:240] Negotiation successful
0927 10:48:20.255834 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1271}
I20260927 10:48:20.256433 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.245293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48070 (local address 127.0.98.62:36311)
0927 10:48:20.245422 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:20.245429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:20.245486 (+ 57us) server_negotiation.cc:407] Connection header received
0927 10:48:20.245629 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:20.245632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:20.245683 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:20.245768 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:20.246395 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.247163 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.247990 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.248336 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.248436 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:20.252351 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:20.252354 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:20.252356 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:20.252379 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:20.254118 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:20.254645 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:20.254649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:20.254651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:20.254740 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:20.255033 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:20.255036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:20.255037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:20.255230 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:20.255435 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:20.256179 (+ 744us) server_negotiation.cc:299] Negotiation successful
0927 10:48:20.256320 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:48:20.256835 23249 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:48:20.260833 23099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f82289dec1b4b1f80e197a78bae2f07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36311 } } peers { permanent_uuid: "8fd4e777df6547d882318ec766f49745" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 } } peers { permanent_uuid: "7b343bb41ef2441aac0a4e614c8f1c46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } }
I20260927 10:48:20.261157 23099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f82289dec1b4b1f80e197a78bae2f07, State: Initialized, Role: FOLLOWER
I20260927 10:48:20.261549 23099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f82289dec1b4b1f80e197a78bae2f07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36311 } } peers { permanent_uuid: "8fd4e777df6547d882318ec766f49745" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 } } peers { permanent_uuid: "7b343bb41ef2441aac0a4e614c8f1c46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } }
I20260927 10:48:20.262097 23256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f82289dec1b4b1f80e197a78bae2f07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36311 } } peers { permanent_uuid: "8fd4e777df6547d882318ec766f49745" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 } } peers { permanent_uuid: "7b343bb41ef2441aac0a4e614c8f1c46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } } }
I20260927 10:48:20.262249 23256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07 [sys.catalog]: This master's current role is: FOLLOWER
Sep 27 10:48:20 dist-test-slave-dzhw krb5kdc[23018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 10:48:20.263453 23099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07 [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 10:48:20 dist-test-slave-dzhw krb5kdc[23018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 10:48:20.267463 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.257596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37013 (local address 127.0.0.1:57244)
0927 10:48:20.257978 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:20.257993 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:20.258067 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:20.258395 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:20.258400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:20.258611 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:20.258729 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:20.258736 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.259777 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.259780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:20.260584 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:20.260597 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.261157 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.261161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:20.261246 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:20.261791 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:20.261814 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:20.263521 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:20.265706 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:20.265710 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:20.265713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:20.265975 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:20.266266 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:20.266269 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:20.266270 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:20.266351 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:20.266944 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:20.266947 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:20.267033 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 10:48:20.267133 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 10:48:20.267278 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260927 10:48:20.268064 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.257674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57244 (local address 127.0.98.60:37013)
0927 10:48:20.257808 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:20.257812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:20.258138 (+ 326us) server_negotiation.cc:407] Connection header received
0927 10:48:20.258181 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:20.258185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:20.258222 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:20.258287 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:20.258853 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.259664 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.260710 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.261029 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.261134 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:20.263692 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:20.263698 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:20.263701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:20.263730 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:20.265530 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:20.266077 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:20.266084 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:20.266086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:20.266169 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:20.266458 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:20.266461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:20.266462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:20.266786 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:20.266912 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:20.267809 (+ 897us) server_negotiation.cc:299] Negotiation successful
0927 10:48:20.267926 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:48:20.268867 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.257859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50734 (local address 127.0.98.61:42327)
0927 10:48:20.257983 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:20.257988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:20.258002 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:48:20.258044 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:20.258046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:20.258098 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:20.258183 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:20.259006 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.259518 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.260635 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.260910 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.260970 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:20.264709 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:20.264714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:20.264717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:20.264756 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:20.266750 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:20.267271 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:20.267275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:20.267277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:20.267368 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:20.267695 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:20.267699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:20.267700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:20.267923 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:20.268194 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:20.268524 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 10:48:20.268669 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 10:48:20.271960 23249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b343bb41ef2441aac0a4e614c8f1c46: Bootstrap starting.
W20260927 10:48:20.272413 23267 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:48:20.272503 23267 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:48:20.270493 23031 master_runner.cc:186] Error getting master registration for 127.0.98.61:42327: OK, instance_id { permanent_uuid: "8fd4e777df6547d882318ec766f49745" instance_seqno: 1790506099939882 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 10:48:20.274940 23031 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:48:20.275436 23249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b343bb41ef2441aac0a4e614c8f1c46: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:20.276046 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:48:20.276494 23249 log.cc:824] T 00000000000000000000000000000000 P 7b343bb41ef2441aac0a4e614c8f1c46: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:20.277526 23073 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8f82289dec1b4b1f80e197a78bae2f07"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48070
I20260927 10:48:20.277778 23073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 10:48:20.277915 23073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:20.279129 23249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b343bb41ef2441aac0a4e614c8f1c46: No bootstrap required, opened a new log
I20260927 10:48:20.279235 23073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f82289dec1b4b1f80e197a78bae2f07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36311 } } peers { permanent_uuid: "8fd4e777df6547d882318ec766f49745" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 } } peers { permanent_uuid: "7b343bb41ef2441aac0a4e614c8f1c46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } }
I20260927 10:48:20.280282 23073 leader_election.cc:288] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07 [CANDIDATE]: Term 1 election: Requested vote from peers 8fd4e777df6547d882318ec766f49745 (127.0.98.61:42327), 7b343bb41ef2441aac0a4e614c8f1c46 (127.0.98.60:37013)
I20260927 10:48:20.284300 23249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b343bb41ef2441aac0a4e614c8f1c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f82289dec1b4b1f80e197a78bae2f07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36311 } } peers { permanent_uuid: "8fd4e777df6547d882318ec766f49745" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 } } peers { permanent_uuid: "7b343bb41ef2441aac0a4e614c8f1c46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } }
I20260927 10:48:20.284659 23249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b343bb41ef2441aac0a4e614c8f1c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b343bb41ef2441aac0a4e614c8f1c46, State: Initialized, Role: FOLLOWER
I20260927 10:48:20.285194 23249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b343bb41ef2441aac0a4e614c8f1c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f82289dec1b4b1f80e197a78bae2f07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36311 } } peers { permanent_uuid: "8fd4e777df6547d882318ec766f49745" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 } } peers { permanent_uuid: "7b343bb41ef2441aac0a4e614c8f1c46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } }
I20260927 10:48:20.286042 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b343bb41ef2441aac0a4e614c8f1c46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f82289dec1b4b1f80e197a78bae2f07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36311 } } peers { permanent_uuid: "8fd4e777df6547d882318ec766f49745" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 } } peers { permanent_uuid: "7b343bb41ef2441aac0a4e614c8f1c46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } } }
I20260927 10:48:20.286487 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b343bb41ef2441aac0a4e614c8f1c46 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:48:20.287060 23249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b343bb41ef2441aac0a4e614c8f1c46 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:48:20.287180 23223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f82289dec1b4b1f80e197a78bae2f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7b343bb41ef2441aac0a4e614c8f1c46"
I20260927 10:48:20.287449 23223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b343bb41ef2441aac0a4e614c8f1c46 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:20.290544 23223 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7b343bb41ef2441aac0a4e614c8f1c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f82289dec1b4b1f80e197a78bae2f07 in term 1.
I20260927 10:48:20.291091 23049 leader_election.cc:302] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b343bb41ef2441aac0a4e614c8f1c46, 8f82289dec1b4b1f80e197a78bae2f07; no voters:
I20260927 10:48:20.291532 23256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:20.291772 23256 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07 [term 1 LEADER]: Becoming Leader. State: Replica: 8f82289dec1b4b1f80e197a78bae2f07, State: Running, Role: LEADER
I20260927 10:48:20.292196 23256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f82289dec1b4b1f80e197a78bae2f07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36311 } } peers { permanent_uuid: "8fd4e777df6547d882318ec766f49745" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 } } peers { permanent_uuid: "7b343bb41ef2441aac0a4e614c8f1c46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } }
I20260927 10:48:20.294582 23256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f82289dec1b4b1f80e197a78bae2f07. Latest consensus state: current_term: 1 leader_uuid: "8f82289dec1b4b1f80e197a78bae2f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f82289dec1b4b1f80e197a78bae2f07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36311 } } peers { permanent_uuid: "8fd4e777df6547d882318ec766f49745" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 } } peers { permanent_uuid: "7b343bb41ef2441aac0a4e614c8f1c46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } } }
I20260927 10:48:20.294718 23256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07 [sys.catalog]: This master's current role is: LEADER
W20260927 10:48:20.295007 23288 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b343bb41ef2441aac0a4e614c8f1c46: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:48:20.295189 23288 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:48:20.301100 23180 master_runner.cc:186] Error getting master registration for 127.0.98.61:42327: OK, instance_id { permanent_uuid: "8fd4e777df6547d882318ec766f49745" instance_seqno: 1790506099939882 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 10:48:20.304945 23180 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:48:20.306367 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.283311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48076 (local address 127.0.98.62:36311)
0927 10:48:20.283470 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:20.283476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:20.285129 (+ 1653us) server_negotiation.cc:407] Connection header received
0927 10:48:20.286468 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:20.286471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:20.286548 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:20.286611 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:20.287631 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.288451 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.292539 (+ 4088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.292844 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.292918 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:20.295497 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:20.295502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:20.295505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:20.295533 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:20.297832 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:20.299387 (+ 1555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:20.299392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:20.299394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:20.299479 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:20.305127 (+ 5648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:20.305132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:20.305134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:20.305428 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:20.305586 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:20.306019 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 10:48:20.306175 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:48:20.307816 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.283385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50740 (local address 127.0.98.61:42327)
0927 10:48:20.283524 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:20.283529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:20.284689 (+ 1160us) server_negotiation.cc:407] Connection header received
0927 10:48:20.284912 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:20.284917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:20.284994 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:20.285113 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:20.289684 (+ 4571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.290443 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.291533 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.291882 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.291962 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:20.296834 (+ 4872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:20.296838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:20.296841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:20.296875 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:20.300270 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:20.305288 (+ 5018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:20.305292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:20.305294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:20.305389 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:20.305731 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:20.305746 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:20.305748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:20.306022 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:20.306171 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:20.307501 (+ 1330us) server_negotiation.cc:299] Negotiation successful
0927 10:48:20.307638 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:48:20.311596 23292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:48:20.314142 23292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:48:20.323604 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.283996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57254 (local address 127.0.98.60:37013)
0927 10:48:20.284136 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:20.284143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:20.307362 (+ 23219us) server_negotiation.cc:407] Connection header received
0927 10:48:20.308571 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:20.308575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:20.308644 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:20.308725 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:20.311320 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.312087 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.313301 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.313575 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.313682 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:20.319357 (+ 5675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:20.319364 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:20.319369 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:20.319409 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:20.321372 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:20.321927 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:20.321930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:20.321932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:20.322031 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:20.322352 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:20.322356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:20.322360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:20.322697 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:20.322862 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:20.323230 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 10:48:20.323389 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:48:20.326597 23223 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7b343bb41ef2441aac0a4e614c8f1c46 [term 1 FOLLOWER]: Refusing update from remote peer 8f82289dec1b4b1f80e197a78bae2f07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:48:20.327467 23289 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b343bb41ef2441aac0a4e614c8f1c46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:48:20.334014 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b343bb41ef2441aac0a4e614c8f1c46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f82289dec1b4b1f80e197a78bae2f07. Latest consensus state: current_term: 1 leader_uuid: "8f82289dec1b4b1f80e197a78bae2f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f82289dec1b4b1f80e197a78bae2f07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36311 } } peers { permanent_uuid: "8fd4e777df6547d882318ec766f49745" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 } } peers { permanent_uuid: "7b343bb41ef2441aac0a4e614c8f1c46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } } }
I20260927 10:48:20.334203 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b343bb41ef2441aac0a4e614c8f1c46 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:48:20.340684 23256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f82289dec1b4b1f80e197a78bae2f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f82289dec1b4b1f80e197a78bae2f07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36311 } } peers { permanent_uuid: "8fd4e777df6547d882318ec766f49745" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 } } peers { permanent_uuid: "7b343bb41ef2441aac0a4e614c8f1c46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } } }
I20260927 10:48:20.340875 23256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:20.341305 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b343bb41ef2441aac0a4e614c8f1c46 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f82289dec1b4b1f80e197a78bae2f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f82289dec1b4b1f80e197a78bae2f07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36311 } } peers { permanent_uuid: "8fd4e777df6547d882318ec766f49745" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 } } peers { permanent_uuid: "7b343bb41ef2441aac0a4e614c8f1c46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } } }
I20260927 10:48:20.341435 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b343bb41ef2441aac0a4e614c8f1c46 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:48:20.341604 23293 mvcc.cc:205] Tried to move back new op lower bound from 7333912986921832448 to 7333912986801487872. Current Snapshot: MvccSnapshot[applied={T|T < 7333912986921832448}]
I20260927 10:48:20.341898 23294 mvcc.cc:205] Tried to move back new op lower bound from 7333912986921832448 to 7333912986801487872. Current Snapshot: MvccSnapshot[applied={T|T < 7333912986921832448}]
I20260927 10:48:20.344609 23292 catalog_manager.cc:1347] Generated new cluster ID: bee6cdf7ab4f4950ac8a0817cbd21258
I20260927 10:48:20.344765 23292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:48:20.350987 23174 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } attempt: 4
Sep 27 10:48:20 dist-test-slave-dzhw krb5kdc[23018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 10:48:20.364707 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.351485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37013 (local address 127.0.0.1:57266)
0927 10:48:20.351688 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:20.351704 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:20.351810 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:20.352173 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:20.352177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:20.352434 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:20.352587 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:20.352595 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.353624 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.353630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:20.354590 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:20.354601 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.356249 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.356253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:20.356349 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:20.357010 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:20.357027 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:20.358893 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:20.361161 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:20.361167 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:20.361170 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:20.361452 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:20.361777 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:20.361780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:20.361782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:20.361868 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:20.364080 (+ 2212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:20.364083 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:20.364198 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 10:48:20.364311 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 10:48:20.364483 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260927 10:48:20.365861 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:20.351532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57266 (local address 127.0.98.60:37013)
0927 10:48:20.351661 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:20.351666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:20.351751 (+ 85us) server_negotiation.cc:407] Connection header received
0927 10:48:20.351895 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:20.351898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:20.351951 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:20.352026 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:20.352712 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.353517 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.354733 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:20.355015 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:20.355131 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:20.359054 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:20.359058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:20.359062 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:20.359090 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:20.361045 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:20.361579 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:20.361584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:20.361586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:20.361673 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:20.361942 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:20.361948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:20.361950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:20.363976 (+ 2026us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:20.364084 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:20.365562 (+ 1478us) server_negotiation.cc:299] Negotiation successful
0927 10:48:20.365697 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:48:20.370206 23174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fd4e777df6547d882318ec766f49745: Bootstrap starting.
I20260927 10:48:20.376945 23174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fd4e777df6547d882318ec766f49745: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:20.379633 23292 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:48:20.381141 23292 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:48:20.378872 23174 log.cc:824] T 00000000000000000000000000000000 P 8fd4e777df6547d882318ec766f49745: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:20.391813 23174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fd4e777df6547d882318ec766f49745: No bootstrap required, opened a new log
I20260927 10:48:20.393983 23292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07: Generated new TSK 0
I20260927 10:48:20.395802 23292 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:48:20.400414 23174 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fd4e777df6547d882318ec766f49745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f82289dec1b4b1f80e197a78bae2f07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36311 } } peers { permanent_uuid: "8fd4e777df6547d882318ec766f49745" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 } } peers { permanent_uuid: "7b343bb41ef2441aac0a4e614c8f1c46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } }
I20260927 10:48:20.400750 23174 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fd4e777df6547d882318ec766f49745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fd4e777df6547d882318ec766f49745, State: Initialized, Role: FOLLOWER
I20260927 10:48:20.401196 23174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fd4e777df6547d882318ec766f49745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f82289dec1b4b1f80e197a78bae2f07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36311 } } peers { permanent_uuid: "8fd4e777df6547d882318ec766f49745" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 } } peers { permanent_uuid: "7b343bb41ef2441aac0a4e614c8f1c46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } }
I20260927 10:48:20.401953 23300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fd4e777df6547d882318ec766f49745 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f82289dec1b4b1f80e197a78bae2f07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36311 } } peers { permanent_uuid: "8fd4e777df6547d882318ec766f49745" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 } } peers { permanent_uuid: "7b343bb41ef2441aac0a4e614c8f1c46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } } }
I20260927 10:48:20.402170 23300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fd4e777df6547d882318ec766f49745 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:48:20.403219 23174 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fd4e777df6547d882318ec766f49745 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:48:20.403780 23148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f82289dec1b4b1f80e197a78bae2f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8fd4e777df6547d882318ec766f49745"
I20260927 10:48:20.403492 23147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fd4e777df6547d882318ec766f49745 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:20.406654 23147 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8fd4e777df6547d882318ec766f49745 [term 1 FOLLOWER]: Refusing update from remote peer 8f82289dec1b4b1f80e197a78bae2f07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:48:20.407866 23289 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fd4e777df6547d882318ec766f49745" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:48:20.408566 23148 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8fd4e777df6547d882318ec766f49745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f82289dec1b4b1f80e197a78bae2f07 in term 1.
I20260927 10:48:20.422719 23315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fd4e777df6547d882318ec766f49745 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f82289dec1b4b1f80e197a78bae2f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f82289dec1b4b1f80e197a78bae2f07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36311 } } peers { permanent_uuid: "8fd4e777df6547d882318ec766f49745" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 } } peers { permanent_uuid: "7b343bb41ef2441aac0a4e614c8f1c46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } } }
I20260927 10:48:20.423053 23315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fd4e777df6547d882318ec766f49745 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:48:20.423367 23300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fd4e777df6547d882318ec766f49745 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f82289dec1b4b1f80e197a78bae2f07. Latest consensus state: current_term: 1 leader_uuid: "8f82289dec1b4b1f80e197a78bae2f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f82289dec1b4b1f80e197a78bae2f07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36311 } } peers { permanent_uuid: "8fd4e777df6547d882318ec766f49745" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 } } peers { permanent_uuid: "7b343bb41ef2441aac0a4e614c8f1c46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } } }
I20260927 10:48:20.423485 23300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fd4e777df6547d882318ec766f49745 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 10:48:20.426746 23311 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8fd4e777df6547d882318ec766f49745: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:48:20.426839 23311 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:48:20.429618 23313 mvcc.cc:205] Tried to move back new op lower bound from 7333912987189661696 to 7333912986801487872. Current Snapshot: MvccSnapshot[applied={T|T < 7333912986921832448}]
W20260927 10:48:20.445916 23179 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:20.443799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50750 (local address 127.0.98.61:42327)
0927 10:48:20.443951 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:20.443956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:20.444613 (+ 657us) server_negotiation.cc:407] Connection header received
0927 10:48:20.444852 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:20.444855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:20.444916 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:20.444977 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:20.445754 (+ 777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50750: BlockingRecv error: recv got EOF from 127.0.0.1:50750 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 10:48:20.448809 23252 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:20.446689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57270 (local address 127.0.98.60:37013)
0927 10:48:20.446834 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:20.446850 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:20.447722 (+ 872us) server_negotiation.cc:407] Connection header received
0927 10:48:20.447937 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:20.447940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:20.448005 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:20.448064 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:20.448659 (+ 595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57270: BlockingRecv error: recv got EOF from 127.0.0.1:57270 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 10:48:20.450438 23103 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:20.447230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48088 (local address 127.0.98.62:36311)
0927 10:48:20.447373 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:20.447379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:20.449397 (+ 2018us) server_negotiation.cc:407] Connection header received
0927 10:48:20.449576 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:20.449580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:20.449646 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:20.449700 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:20.450260 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48088: BlockingRecv error: recv got EOF from 127.0.0.1:48088 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 10:48:20.455721 23179 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:20.454021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50766 (local address 127.0.98.61:42327)
0927 10:48:20.454162 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:20.454167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:20.454627 (+ 460us) server_negotiation.cc:407] Connection header received
0927 10:48:20.454875 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:20.454882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:20.454979 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:20.455042 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:20.455597 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50766: BlockingRecv error: recv got EOF from 127.0.0.1:50766 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 10:48:20.461320 23179 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:20.459736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50772 (local address 127.0.98.61:42327)
0927 10:48:20.459857 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:20.459861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:20.460368 (+ 507us) server_negotiation.cc:407] Connection header received
0927 10:48:20.460550 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:20.460553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:20.460609 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:20.460667 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:20.461208 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50772: BlockingRecv error: recv got EOF from 127.0.0.1:50772 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 10:48:20.468653 23179 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:20.466907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50774 (local address 127.0.98.61:42327)
0927 10:48:20.467028 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:20.467033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:20.467588 (+ 555us) server_negotiation.cc:407] Connection header received
0927 10:48:20.467784 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:20.467787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:20.467864 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:20.467921 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:20.468518 (+ 597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50774: BlockingRecv error: recv got EOF from 127.0.0.1:50774 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 10:48:20.477504 23179 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:20.475579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50776 (local address 127.0.98.61:42327)
0927 10:48:20.475725 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:20.475730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:20.476311 (+ 581us) server_negotiation.cc:407] Connection header received
0927 10:48:20.476551 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:20.476556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:20.476621 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:20.476684 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:20.477354 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50776: BlockingRecv error: recv got EOF from 127.0.0.1:50776 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 10:48:20.491535 23179 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:20.489461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50780 (local address 127.0.98.61:42327)
0927 10:48:20.489626 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:20.489632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:20.490305 (+ 673us) server_negotiation.cc:407] Connection header received
0927 10:48:20.490591 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:20.490597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:20.490669 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:20.490728 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:20.491379 (+ 651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50780: BlockingRecv error: recv got EOF from 127.0.0.1:50780 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 10:48:20.512384 23179 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:20.510871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50796 (local address 127.0.98.61:42327)
0927 10:48:20.511044 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:20.511049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:20.511275 (+ 226us) server_negotiation.cc:407] Connection header received
0927 10:48:20.511445 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:20.511449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:20.511543 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:20.511676 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:20.512224 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50796: BlockingRecv error: recv got EOF from 127.0.0.1:50796 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 10:48:20.548700 23179 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:20.547200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50808 (local address 127.0.98.61:42327)
0927 10:48:20.547447 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:20.547454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:20.547573 (+ 119us) server_negotiation.cc:407] Connection header received
0927 10:48:20.547743 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:20.547746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:20.547807 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:20.547891 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:20.548526 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50808: BlockingRecv error: recv got EOF from 127.0.0.1:50808 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 10:48:20.616704 23179 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:20.615227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50824 (local address 127.0.98.61:42327)
0927 10:48:20.615482 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:20.615494 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:20.615641 (+ 147us) server_negotiation.cc:407] Connection header received
0927 10:48:20.615855 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:20.615859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:20.615926 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:20.616027 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:20.616538 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50824: BlockingRecv error: recv got EOF from 127.0.0.1:50824 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
W20260927 10:48:20.748921 23179 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:20.747440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50826 (local address 127.0.98.61:42327)
0927 10:48:20.747664 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:20.747670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:20.747810 (+ 140us) server_negotiation.cc:407] Connection header received
0927 10:48:20.748026 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:20.748035 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:20.748095 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:20.748198 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:20.748744 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50826: BlockingRecv error: recv got EOF from 127.0.0.1:50826 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 10:48:20.944650 23371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f82289dec1b4b1f80e197a78bae2f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f82289dec1b4b1f80e197a78bae2f07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36311 } } peers { permanent_uuid: "8fd4e777df6547d882318ec766f49745" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42327 } } peers { permanent_uuid: "7b343bb41ef2441aac0a4e614c8f1c46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37013 } } }
I20260927 10:48:20.944926 23371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f82289dec1b4b1f80e197a78bae2f07 [sys.catalog]: This master's current role is: LEADER
W20260927 10:48:21.009033 23179 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:21.007731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50836 (local address 127.0.98.61:42327)
0927 10:48:21.007899 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:21.007903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:21.007922 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:48:21.008127 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:21.008130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:21.008213 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:21.008308 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:21.008905 (+ 597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50836: BlockingRecv error: recv got EOF from 127.0.0.1:50836 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 10:48:21.296672 23288 catalog_manager.cc:1259] Loaded cluster ID: bee6cdf7ab4f4950ac8a0817cbd21258
I20260927 10:48:21.296787 23288 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7b343bb41ef2441aac0a4e614c8f1c46: loading cluster ID for follower catalog manager: success
I20260927 10:48:21.298681 23288 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7b343bb41ef2441aac0a4e614c8f1c46: acquiring CA information for follower catalog manager: success
I20260927 10:48:21.299952 23288 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7b343bb41ef2441aac0a4e614c8f1c46: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:48:21.428242 23311 catalog_manager.cc:1259] Loaded cluster ID: bee6cdf7ab4f4950ac8a0817cbd21258
I20260927 10:48:21.428335 23311 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8fd4e777df6547d882318ec766f49745: loading cluster ID for follower catalog manager: success
I20260927 10:48:21.429956 23311 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8fd4e777df6547d882318ec766f49745: acquiring CA information for follower catalog manager: success
I20260927 10:48:21.431257 23311 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8fd4e777df6547d882318ec766f49745: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:48:21.528604 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:21.524038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50840 (local address 127.0.98.61:42327)
0927 10:48:21.524366 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:21.524374 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:21.524400 (+ 26us) server_negotiation.cc:407] Connection header received
0927 10:48:21.524617 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:21.524621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:21.524797 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:21.524938 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:48:21.525249 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:21.525846 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:21.526797 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:21.527047 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:21.527166 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:21.527540 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:48:21.528130 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:48:21.528281 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:21.528331 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 10:48:21.528400 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":97,"threads_started":1}
I20260927 10:48:21.537117 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:21.532603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57274 (local address 127.0.98.60:37013)
0927 10:48:21.532917 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:21.532922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:21.532938 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:48:21.533161 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:21.533164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:21.533324 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:21.533481 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:48:21.533712 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:21.534293 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:21.535195 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:21.535443 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:21.535568 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:21.535901 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:48:21.536506 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:48:21.536807 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:21.536848 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 10:48:21.536917 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":79,"threads_started":1}
I20260927 10:48:21.547257 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:21.541468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57290 (local address 127.0.98.60:37013)
0927 10:48:21.541624 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:21.541629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:21.542634 (+ 1005us) server_negotiation.cc:407] Connection header received
0927 10:48:21.542778 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:21.542782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:21.542856 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:21.542930 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:48:21.543274 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:21.544056 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:21.545667 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:21.546005 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:21.546082 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:21.546664 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:48:21.546754 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:48:21.546836 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:21.546983 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 10:48:21.547063 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:48:21.547612 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:21.541312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48104 (local address 127.0.98.62:36311)
0927 10:48:21.541800 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:21.541806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:21.542186 (+ 380us) server_negotiation.cc:407] Connection header received
0927 10:48:21.542445 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:21.542449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:21.542676 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:21.542842 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:48:21.543151 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:21.544056 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:21.545663 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:21.546008 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:21.546082 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:21.546634 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:48:21.547080 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:48:21.547154 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:21.547341 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 10:48:21.547413 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":77,"threads_started":1}
I20260927 10:48:21.547752 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:21.541336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.0.98.61:42327)
0927 10:48:21.543225 (+ 1889us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:21.543230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:21.543247 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:48:21.543311 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:21.543313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:21.543371 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:21.543462 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:48:21.543883 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:21.544524 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:21.545610 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:21.545938 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:21.546071 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:21.546344 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:48:21.546480 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:48:21.546551 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:21.547528 (+ 977us) server_negotiation.cc:299] Negotiation successful
0927 10:48:21.547603 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1766}
I20260927 10:48:21.551573 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 23031
I20260927 10:48:21.559206 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 23106
I20260927 10:48:21.566300 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkqkM3n/build/debug/bin/kudu with pid 23180
2026-09-27T10:48:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3363 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6778 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (289425 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST